SlowGuess/NAACL_2022_OCR huggingface dataset:
We use joint accuracy and slot accuracy as our evaluation metrics. Joint accuracy is the proportion of dialogue turns where the value of each slot is correctly predicted. Slot accuracy only considers individual slot-level accuracy. The ground-truth of slot value is set to none if the slot has not been mentioned in dialogue.
Same as the previous work (Shan et al., 2020; Ye et al., 2021), we use BERT-base-uncased model as
the encoders of LUNA where only the utterance encoder is fine-tuned and the parameters of the other two encoders are fixed. BERT-base has 12 layers of 784 hidden units and 12 self-attention heads. The number of attention heads in multi-head attention in our alignment module is set to 4. The number of layers in slot self-attention and turn self-attention is set to 4 and 2 respectively. During the training process, we use Adam optimizer (Kingma and Ba, 2015) and set the warmup proportion to 0.1. Considering that the encoder is a pre-trained BERT model while the other parts in our model needs to be trained from scratch, we use different learning rates for those parts. Specifically, the peak learning rate is set to 3e-5 for the utterance encoder and 1e-4 for the remaining parts. The maximum input sequence length in BERT is set to 512. For MultiWOZ 2.0, MultiWOZ 2.1, and MultiWOZ 2.2, we apply the same hyperparameter settings.
Table 2 shows the joint accuracy and the slot accuracy of our model and other baselines on the test sets of MultiWOZ 2.0, 2.1, and 2.2, where some models are not tested on the 2.2 version since it was released shortly. As shown in the table, among the models without data augmentation, our model LUNA achieves state-of-the-art performance on these datasets with joint accuracy of  $55.31\%$ ,  $57.62\%$ , and  $56.13\%$ , which has a measurable im
Figure 2: Joint accuracy at every turn.
provement (0.97%, 1.26%, and 4.43%) over the previous best results, illustrating the effectiveness of slot-turn alignment in DST task.
It can be observed that the three data augmented methods reach higher than  $58\%$  joint accuracy on MultiWOZ 2.1. We believe that these data augmentation skills are versatile. If they can improve the results of TripPy that lags behind our model, we reasonably speculate that these skills can also improve the effect of LUNA. Besides, all these models are based upon TripPy, which employs a label map as extra supervision. The label map is a dictionary of synonyms, which is used during the testing phase. For example, the official label of the slot "hotel-area" in ontology is "centre". But label map regards all its synonyms, such as "center", are also the ground truth. We think that this manner severely reduces the difficulty of the DST task. As shown in Table 2, the performance of TripPy degrades dramatically when the label map is removed. By contrast, our model does not rely on any extra information and is more generalized.
Additionally, Table 2 lists the number of trainable parameters of some baselines and our model, which illustrates that our alignment module containing multiple self-attention does not introduce large model parameters. Compared with the baselines, the size of our model is comparable.
Accuracy at Every Turn. In practice, the dialogue states of longer dialogues tend to be more difficult to be correctly predicted as the model needs to consider more dialogue history. In this section, we further analyze the relationship between the depth of conversation and the prediction accuracy. The joint accuracy at every turn of TripPy, STAR, and LUNA on MultiWOZ 2.1 test set is shown in Figure 2. It presents that the scores of LUNA and STAR are basically the same when the number of conversation turns is less than 3. While as the conversation turns increases from 3, the superiority of LUNA gradu
ally becomes obvious. This is because that both TripPy and STAR exploit all dialogue utterances to assign value for each slot. This may introduce more useless information that causes confusion to the current slots. Whereas, LUNA only uses the most relevant utterance to assign slot value, which avoids interference by useless information.
5.2 Ablation on Alignment Module
To explore the effectiveness of each part in our proposed alignment module, we conduct an ablation study of these parts on the test set of MultiWOZ 2.1, as shown in Table 3.
Model	Align Acc	Joint Acc.
LUNA	97.50	57.62
- Alignment module	-	53.46 (-4.16)
- Overall slot-to-turn alignment	95.23 (-2.27)	54.70 (-2.92)
- Auxiliary task	96.30 (-1.20)	55.29 (-2.33)
Table 3: The ablation study of the alignment module on the MultiWOZ 2.1. Alignment accuracy  $(\%)$  is defined as the ratio of dialogue for which the utterance turn of each slot is correctly aligned.
First, we remove the whole alignment module and only use the representations of slots obtained by token-slot attention Eq.5 to match the value. The results show that model performance has dropped a lot (4.16 joint accuracy), proving that there are many useless tokens in the conversation history, which interfere the prediction accuracy of slot value. Next, we remove the layer of overall slot-to-turn alignment. We can see that this also severely damages the model performance on both alignment accuracy and joint accuracy. This illustrates that it is not enough to only use the information of a single slot for the alignment. The model needs to comprehensively consider all slots information, such as semantic correlations and temporal correlations among slots to accurately align slots and dialogue turns. Finally, we remove the auxiliary ranking task and the results decrease by 1.20 on alignment accuracy and 2.33 on joint accuracy. This proves that the temporal correlation among slots is important in our model which could facilitate the alignment, as we explained in the section of Introduction. More intuitive explanations will be given in the next section through an example of visualization.
Hard or Soft Alignment. From Table 3, although our hard alignment is highly accurate (Acc 97.50), we should further explore whether it can be replaced by a soft alignment to avoid the risk of error propagation. Whereupon, we design a soft align
Turn1 User : I am looking for Corsica restaurants in the east of Cambridge.
Turn2 Sys: Unfortunately there is not a corsica restaurant in the east of Cambridge, is there another type of food you would like to look for? User: I'll think about it. I would like a moderately priced restaurant.
Turn3 Sys : How about curry prince? It is a moderately priced Indian restaurant. User : Curry prince sounds terrific.
Turn4 Sys:Would you like me to make you a reservation?User:Yesplease,for7at11:45on Thursday.
(c)
Figure 3: Visualization of LUNA on an example from MultiWOZ 2.1, which is a process of predicting the value to slot "restaurant-food" at turn-4. "R-" in figure denotes "restaurant-". The golden value of "restaurant-food" is "Indian" and the confusion value is "Corsica". (a) is the distribution of Token-Slot Attention calculated by Eq.5 where the columns are the four heads in multi-head attention. (b) is the softmax distribution of alignment over turns calculated by Eq.12. (c)(d) are the distributions of Slot-Turn Attention calculated by Eq.10 where (d) is the version after removing ranking loss.
Embedding Trained w/o Ranking
Figure 4: Visualization of slot embedding using t-sne. Each point represents the slot aligned to the corresponding turn. We plot 100 slot embeddings for each turn.
Embedding Trained w/ Ranking
ment that is a weighted sum of all turns with the alignment distribution over the turns as weights (Eq. 12). The experimental results show that compared with hard alignment, the Joint accuracy of soft alignment on MultiWOZ 2.1 drops to 57.53 (-0.09). The reason is that the soft alignment encounters the problem of noises introduced from irrelevant utterances. In other words, risk of error propagation and noise-avoiding are a trade-off. The experimental results show that the benefits of our proposed hard alignment outweigh the risk.
Figure 3 gives an example to visualize the process of predicting the value to slot "restaurant-food". In this example, the golden slot value is "Indian" and "Corsica" is the confusion value. As shown in sub-figure (a), the slot assigns high attention weight in all heads to both "Indian" and "Corsica", because as of this step, it cannot determine which one is the correct value. At the last step, after
the bi-directional fusion in our proposed alignment module, the model successfully assigns turn-3 a larger alignment score than turn-1, as shown in subfigure (b). In other words, the model has realized that the utterance of turn-3 (containing "Indian") is more important than turn-1 (containing "Corsica"). This can avoid the confusion caused by "Corsica".
We next analyze sub-figures (c) and (d). As we can see, all turns focus on the slot "restaurant-food" as they incorporates its single slot information through Eq.8. For the column of turn-3, if the model is supervised with the auxiliary ranking task, it will also consider the information of "restaurant-book day" and "restaurant-book time". Sub-figure (b) indicates that these two slots are easy-aligned (with turn-4). Meanwhile, the model learns that the order of the three slots is ["restaurant-food", "restaurant-book day", "restaurant-book time"]. Thereby, the alignment of "restaurant-food" and turn-3 becomes easier.
Figure 4 displays the 2-d visualization of slot embeddings obtained by Eq. 7. It can be seen that without ranking loss, the slot representations are irregular and borderless. Under the supervision of the ranking loss, the model can learn the boundaries between the slots aligned with different turns.
In this work, we reveal the problem in DST that exploiting all dialogue utterances to assign value to slots may cause suboptimal results. To alleviate it, we propose LUNA, a slot-turn alignment
enhanced approach. and design a ranking-based auxiliary task to supervise LUNA to learn the temporal correlations among slots. Comprehensive experiments are conducted on MultiWOZ 2.0, 2.1, and 2.2 and the results show that LUNA achieves new state-of-the-art results. Moreover, the visualization demonstrates the interpretability of LUNA.
7 Acknowledge
We would like to thank the anonymous reviewers for their useful feedback. This work is supported by the National Key Research and Development Program of China under Grant No. 2020AAA0108600.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a largescale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Junfan Chen, Richong Zhang, Yongyi Mao, and Jie Xu. 2020a. Parallel interactive networks for multi-domain dialogue state generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1921-1931, Online. Association for Computational Linguistics.
Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020b. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7521-7528. AAAI Press.
Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021. Preview, attend and review: Schema-aware curriculum learning for multi-domain dialog state tracking. ArXiv preprint, abs/2106.00291.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tür. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines.
Yue Feng, Yang Wang, and Hang Li. 2021. A sequence-to-sequence approach to dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1714-1725, Online. Association for Computational Linguistics.
Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35-44, 1st virtual meeting. Association for Computational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jiaying Hu, Yan Yang, Chencai Chen, Liang He, and Zhou Yu. 2020. SAS: Dialogue state tracking via slot attention and slot information sharing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6366-6375, Online. Association for Computational Linguistics.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567-582, Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. SUMBT: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483, Florence, Italy. Association for Computational Linguistics.
Shiyang Li, Semih Yavuz, Kazuma Hashimoto, Jia Li, Tong Niu, Nazneen Rajani, Xifeng Yan, Yingbo Zhou, and Caiming Xiong. 2020. Coco: Controllable counterfactuals for evaluating dialogue state trackers. ArXiv preprint, abs/2010.12850.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020. Dialogue: A natural language understanding benchmark for task-oriented dialogue. ArXiv preprint, abs/2009.13570.
Nikola Mrkšić, Diarmuid Řeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788, Vancouver, Canada. Association for Computational Linguistics.
Jun Quan and Deyi Xiong. 2020. Modeling long context for task-oriented dialogue state generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7119-7124, Online. Association for Computational Linguistics.
Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2780-2786, Brussels, Belgium. Association for Computational Linguistics.
Yong Shan, Zekang Li, Jinchao Zhang, Fandong Meng, Yang Feng, Cheng Niu, and Jie Zhou. 2020. A contextual hierarchical attention network with adaptive objective for dialogue state tracking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6322-6333, Online. Association for Computational Linguistics.
Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022. Bort: Back and denoising reconstruction for end-to-end task-oriented dialog. In *Findings of the Association for Computational Linguistics: NAACL* 2022. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Jason D Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4-33.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence, Italy. Association for Computational Linguistics.
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Machine Learning,
Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 1192-1199. ACM.
Kaige Xie, Cheng Chang, Liliang Ren, Lu Chen, and Kai Yu. 2018. Cost-sensitive active learning for dialogue state tracking. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 209-213, Melbourne, Australia. Association for Computational Linguistics.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448-1457, Melbourne, Australia. Association for Computational Linguistics.
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021. Slot self-attentive dialogue state tracking. In Proceedings of the Web Conference 2021, pages 1598-1608.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 154-167, Barcelona, Spain (Online). Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458–1467, Melbourne, Australia. Association for Computational Linguistics.
Su Zhu, Jieyu Li, Lu Chen, and Kai Yu. 2020. Efficient context and schema fusion networks for multi-domain dialogue state tracking. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 766-781, Online. Association for Computational Linguistics.