SlowGuess/ACL_2020_OCR huggingface dataset:
HJST is proposed together with FJST; it encodes the dialogue history using an LSTM like FJST but uses a hierarchical network (Eric et al., 2019).
SUMBT exploits BERT-base as the encoder for the dialogue context and slot-value pairs. After encoding them, it scores every candidate slot-value pair in a non-parametric manner using a distance measure (Lee et al., 2019).
HyST employs a hierarchical RNN encoder and takes a hybrid approach that incorporates both a predefined ontology-based setting and an open vocabulary-based setting (Goel et al., 2019).
DST Reader formulates the problem of DST as an extractive QA task; it uses BERT-base to make the contextual word embeddings and extracts the value of the slots from the input as a span (Gao et al., 2019).
TRADE encodes the whole dialogue context with a bidirectional GRU and decodes the value for every slot using a copy-augmented GRU decoder (Wu et al., 2019).
COMER uses BERT-large as a feature extractor and a hierarchical LSTM decoder to generate the current turn dialogue state itself as the target sequence (Ren et al., 2019).
NADST uses a Transformer-based non-autoregressive decoder to generate the current turn dialogue state (Le et al., 2020b).
ML-BST uses a Transformer-based architecture to encode the dialogue context with the domain and slot information and combines the outputs in a late fusion approach. Then, it generates the slot values and the system response jointly (Le et al., 2020a).
DS-DST uses two BERT-base encoders and takes a hybrid approach of predefined ontology-based DST and open vocabulary-based DST. It defines picklist-based slots for classification similarly to SUMBT and span-based slots for span extraction like DST Reader (Zhang et al., 2019).
<sup>6</sup>The excluded domains take up only a small portion of the dataset and do not even appear in the test set.  
<sup>7</sup>github.com/jasonwu0731/trade-dst  
<sup>8</sup>github.com/huggingface/transformers
Table 1: Joint goal accuracy on the test set of MultiWOZ 2.0 and 2.1. * indicates a result borrowed from Eric et al. (2019). HyST and DS-DST use a hybrid approach, partially taking advantage of the predefined ontology. † indicates the case where BERT-large is used for our model.
MultiWOZ 2.0	MultiWOZ 2.1
Predefined Ontology
HJST* (Eric et al., 2019)	38.40	35.55
FJST* (Eric et al., 2019)	40.20	38.00
SUMBT (Lee et al., 2019)	42.40	-
HyST* (Goel et al., 2019)	42.33	38.10
DS-DST (Zhang et al., 2019)	-	51.21
DST-picklist (Zhang et al., 2019)	-	53.30
Open Vocabulary
DST Reader* (Gao et al., 2019)	39.41	36.40
TRADE* (Wu et al., 2019)	48.60	45.60
COMER (Ren et al., 2019)	48.79	-
NADST (Le et al., 2020b)	50.52	49.04
ML-BST (Le et al., 2020a)	-	50.91
SOM-DST (ours)	51.72	53.01
SOM-DST† (ours)	52.32	53.68
DST-picklist is proposed together with DS-DST and uses a similar architecture, but it performs only predefined ontology-based DST considering all slots as picklist-based slots (Zhang et al., 2019).
5.1 Joint Goal Accuracy
Table 1 shows the joint goal accuracy of SOM-DST and other models on the test set of MultiWOZ 2.0 and MultiWOZ 2.1. Joint goal accuracy is an accuracy which checks whether all slot values predicted at a turn exactly match the ground truth values.
As shown in the table, SOM-DST achieves state-of-the-art performance in an open vocabulary-based setting. Interestingly, on the contrary to the previous works, our model achieves higher performance on MultiWOZ 2.1 than on MultiWOZ 2.0. This is presumably because our model, which explicitly uses the dialogue state labels as input, benefits more from the error correction on the state annotations done in MultiWOZ 2.1.[9]
Table 2: Domain-specific results on the test set of Multi-WOZ 2.1. Our model outperforms other models in taxi and train domains.
Domain	Model	Joint Accuracy	Slot Accuracy
Attraction	NADST	66.83	98.79
ML-BST	70.78	99.06
SOM-DST (ours)	69.83	98.86
Hotel	NADST	48.76	97.70
ML-BST	49.52	97.50
SOM-DST (ours)	49.53	97.35
Restaurant	NADST	65.37	98.78
ML-BST	66.50	98.76
SOM-DST (ours)	65.72	98.56
Taxi	NADST	33.80	96.69
ML-BST	23.05	96.42
SOM-DST (ours)	59.96	98.01
Train	NADST	62.36	98.36
ML-BST	65.12	90.22
SOM-DST (ours)	70.36	98.67
5.2 Domain-Specific Accuracy
Table 2 shows the domain-specific results of our model and the concurrent works which report such results (Le et al., 2020a,b). Domain-specific accuracy is the accuracy measured on a subset of the predicted dialogue state, where the subset consists of the slots specific to a domain.
While the performance is similar to or a little lower than that of other models in other domains, SOM-DST outperforms other models in taxi and train domains. This implies that the state-of-the-art joint goal accuracy of our model on the test set comes mainly from these two domains.
A characteristic of the data from these domains is that they consist of challenging conversations; the slots of these domains are filled with more diverse values than other domains,[10] and there are more than one domain changes, i.e., the user changes the conversation topic during a dialogue more than once. For a specific example, among the dialogues where the domain switches more than once, the number of conversations that end in taxi domain is ten times more than in other cases. A more detailed statistics are given in Table 10 in Appendix A.
Therefore, we assume our model performs relatively more robust DST in such challenging conversations. We conjecture that this strength attributes to the effective utilization of the previous turn dialogue state in its explicit form, like using a memory;
10The statistics of the slot value vocabulary size are shown in Table 9 in Appendix A.
$^{9}$ Eric et al. (2019) report that the correction of the annotations done in MultiWOZ 2.1 changes about  $32\%$  of the state annotations of MultiWOZ 2.0, which indicates that MultiWOZ 2.0 consists of many annotation errors.
Table 3: Joint goal accuracy on the MultiWOZ 2.1 test set when the four-way state operation prediction changes to two-way, three-way, or six-way.
State Operations	Joint Accuracy
4	CARRYOVER, DELETE, DONTCARE, UPDATE	53.01
2	CARRYOVER, NON-CARRYOVER	52.06
3	CARRYOVER, DONTCARE, UPDATE	52.63
3	CARRYOVER, DELETE, UPDATE	52.64
6	CARRYOVER, DELETE, DONTCARE, UPDATE, YES, NO	52.97
the model can explicitly keep even the information mentioned near the beginning of the conversation and directly copy the values from this memory whenever necessary. Figure 1 shows an example of a complicated conversation in MultiWOZ 2.1, where our model accurately predicts the dialogue state. More sample outputs of SOM-DST are provided in Appendix C.
6.1 Choice of State Operations
Table 3 shows the joint goal accuracy where the four-way state operation prediction changes to two-way, three-way, or six-way.
The joint goal accuracy drops when we use two-way state operation prediction, which is a binary classification of whether to (1) carry over the previous slot value to the current turn or (2) generate a new value, like Gao et al. (2019). We assume the reason is that it is better to separately model operations DELETE, DONTCARE, and UPDATE that correspond to the latter class of the binary classification, since the values of DELETE and DONTCARE tend to appear implicitly while the values for UPDATE are often explicitly expressed in the dialogue.
We also investigate the performance when only three operations are used or two more state operations, YES and NO, are used. YES and NO represent the cases where yes or no should be filled as the slot value, respectively. The performance drops in all of the cases.
Table 4 shows the joint goal accuracy of the combinations of the cases where the ground truth is used or not for each of the previous turn dialogue state, state operations at the current turn, and slot
Table 4: Joint goal accuracy of the current and the ground truth-given situations. Relative error rate is the proportion of the error when  $100\%$  is set as the error where no ground truth is used for SOP and SVG. (GT: Ground Truth, SOP: State Operation Prediction, SVG: Slot Value Generation, Pred: Predicted)
GT SOP	GT SVG	Joint Accuracy	Relative Error Rate
Pred Bt-1(w/ Error Propagation)	✓	✓	53.01	100.0
56.37	92.85
89.85	21.60
✓	✓	100.0	0.00
GT Bt-1(w/o Error Propagation)	✓	✓	81.00	100.0
82.80	90.53
96.27	19.63
✓	✓	100.0	0.00
values for UPDATE at the current turn. From this result, we analyze which of state operation predictor and slot value generator is more responsible for the error in the joint goal prediction, under the cases where error propagation occurs or not.
Among the absolute error of  $46.99\%$  made under the situation that error propagation occurs, i.e., the dialogue state predicted at the previous turn is fed to the model, it could be argued that  $92.85\%$  comes from state operation predictor,  $21.6\%$  comes from slot value generator, and  $14.45\%$  comes from both of the components. This indicates that at least  $78.4\%$  to  $92.85\%$  of the error comes from state operation predictor, and at least  $7.15\%$  to  $21.6\%$  of the error comes from slot value generator.[11]
Among the absolute error of  $19\%$  made under the error propagation-free situation, i.e., ground truth previous turn dialogue state is fed to the model, it could be argued that  $90.53\%$  comes from state operation predictor,  $19.63\%$  comes from slot value generator, and  $10.16\%$  comes from both of the components. This indicates that at least  $80.37\%$  to  $90.53\%$  of the error comes from state operation predictor, and at least  $9.47\%$  to  $19.63\%$  of the error comes from slot value generator.
Error propagation that comes from using the dialogue state predicted at the previous turn increases the error 2.47  $(= \frac{100 - 53.01}{100 - 81.00})$  times. Both with and without error propagation, a relatively large amount
<sup>11</sup>The calculation of the numbers in the paragraph is done as follows. (The figures in the paragraph immediately below are calculated in the same way.)
$100 - 53.01 = 46.99$ $92.85 + 21.6 - 100 = 14.45$
$(100 - 56.37) / 46.99 = 92.85$ $92.85 - 14.45 = 78.4$
$(100 - 89.85) / 46.99 = 21.6$ $21.6 - 14.45 = 7.15$
Table 5: Statistics of the number of state operations and the corresponding F1 scores of our model in MultiWOZ 2.1.
Operation Type	# Operations	F1 score
Train	Valid	Test	Test
CARRYOVER	1,584,757	212,608	212,297	98.66
UPDATE	61,628	8,287	8,399	80.10
DONTCARE	1,911	155	235	32.51
DELETE	1,224	80	109	2.86
Table 6: The minimum, average, and maximum number of slots whose values are generated at a turn, calculated on the test set of MultiWOZ 2.1.
Model	Min #	Avg #	Max #
TRADE	30	30	30
ML-BST	30	30	30
COMER	0	5.72	18
SOM-DST (ours)	0	1.14	9
Table 7: Average inference time per dialogue turn of MultiWOZ 2.1 test set, measured on Tesla V100 with a batch size of 1.  $\dagger$  indicates the case where BERT-large is used for our model.
Model	Joint Accuracy	Latency
TRADE	45.60	340 ms
NADST	49.04	26 ms
SOM-DST (ours)	53.01	27 ms
SOM-DST† (ours)	53.68	40 ms
of error comes from state operation predictor, implying that a large room for improvement currently exists in this component. Improving the state operation prediction accuracy, e.g., by tackling the class imbalance shown in Table 5, may have the potential to increase the overall DST performance by a large margin.
6.3 Efficiency Analysis
In Table 6, we compare the number of slot values generated at a turn among various open vocabulary-based DST models that use an autoregressive decoder.
The maximum number of slots whose values are generated by our model at a turn, i.e., the number of slots on which UPDATE should be performed, is 9 at maximum and only 1.14 on average in the test set of MultiWOZ 2.1.
On the other hand, TRADE and ML-BST generate the values of all the 30 slots at every turn of a dialogue. COMER generates only a subset of the slot values like our model, but it generates the val
ues of all the slots that have a non-NULL value at a turn, which is 18 at maximum and 5.72 on average.
Table 7 shows the latency of SOM-DST and several other models. We measure the inference time for a dialogue turn of MultiWOZ 2.1 on Tesla V100 with a batch size of 1. The models used for comparison are those with official public implementations.
It is notable that the inference time of SOM-DST is about 12.5 times faster than TRADE, which consists of only two GRUs. Moreover, the latency of SOM-DST is compatible with that of NADST, which explicitly uses non-autoregressive decoding, while SOM-DST achieves much higher joint goal accuracy. This shows the efficiency of the proposed selectively overwriting mechanism of SOM-DST, which generates only the minimal slot values at a turn.
In Appendix B, we also investigate Inference Time Complexity (ITC) proposed in the work of Ren et al. (2019), which defines the efficiency of a DST model using  $J$ , the number of slots, and  $M$ , the number of values of a slot.
We propose SOM-DST, an open vocabulary-based dialogue state tracker that regards dialogue state as an explicit memory that can be selectively overwritten. SOM-DST decomposes dialogue state tracking into state operation prediction and slot value generation. This setup makes the generation process efficient because the values of only a minimal subset of the slots are generated at each dialogue turn. SOM-DST achieves state-of-the-art joint goal accuracy on both MultiWOZ 2.0 and MultiWOZ 2.1 datasets in an open vocabulary-based setting. SOM-DST effectively makes use of the explicit dialogue state and discrete operations to perform relatively robust DST even in complicated conversations. Further analysis shows that improving state operation prediction has the potential to increase the overall DST performance dramatically. From this result, we propose that tackling DST with our proposed problem definition is a promising future research direction.
The authors would like to thank the members of Clova AI for proofreading this manuscript.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explorations Newsletter, 19(2):25-35.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014a. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP.
Kyunghyun Cho, Bart Van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014b. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. In SIGDIAL.
Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. In Interspeech.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In SIGDIAL.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwooong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. 2018. Nsml: Meet the mlaas platform with a real-world case study. arXiv preprint arXiv:1810.09957.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Hung Le, Doyen Sahoo, Chenghao Liu, Nancy F. Chen, and Steven C.H. Hoi. 2020a. End-to-end multidomain task-oriented dialogue systems with multilevel neural belief tracker. In Submitted to ICLR 2020.
Hung Le, Richard Socher, and Steven C.H. Hoi. 2020b. Non-autoregressive dialog state tracking. In ICLR.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In ACL.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447, Melbourne, Australia. Association for Computational Linguistics.
Nikola Mrksic, Diarmuid Šeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In ACL.
Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. In 2nd Conversational AI workshop on NeurIPS 2018.
Liliang Ren, Jianmo Ni, and Julian McAuley. 2019. Scalable and accurate dialogue state tracking via hierarchical sequence generation. In EMNLP-IJCNLP.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, et al. 2017. Nsml: A machine learning platform that enables you to focus on your models. arXiv preprint arXiv:1712.05902.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In EACL.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In ACL.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In ACL.
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Philip S. Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In ACL.
A Data Statistics
Table 8: Data Statistics of MultiWOZ 2.1.
Domain	Slots	# of Dialogues	# of Turns
Train	Valid	Test	Train	Valid	Test
Attraction	area, name, type	2,717	401	395	8,073	1,220	1,256
Hotel	price range, type, parking, book stay, book day, book people, area, stars, internet, name	3,381	416	394	14,793	1,781	1,756
Restaurant	food, price range, area, name, book time, book day, book people	3,813	438	437	15,367	1,708	1,726
Taxi	leave at, destination, departure, arrive by	1,654	207	195	4,618	690	654
Train	destination, day, departure, arrive by, book people, leave at	3,103	484	494	12,133	1,972	1,976
Table 9: Statistics of the slot value vocabulary size in MultiWOZ 2.1.
Slot Name	Slot Value Vocabulary Size
Train	Valid	Test
taxi-destination	373	213	213
taxi-departure	357	214	203
restaurant-name	202	162	162
attraction-name	186	145	149
train-leaveat	146	69	117
train-arriveby	112	64	101
restaurant-food	111	81	70
taxi-leaveat	105	68	65
hotel-name	93	65	58
restaurant-book time	64	50	51
taxi-arriveby	95	49	46
train-destination	27	25	24
train-departure	34	23	23
attraction-type	31	17	17
train-book people	11	9	9
hotel-book people	8	8	8
restaurant-book people	9	8	8
hotel-book day	13	7	7
hotel-stars	9	7	7
restaurant-book day	10	7	7
train-day	8	7	7
attraction-area	7	6	6
hotel-area	7	6	6
restaurant-area	7	6	6
hotel-book stay	10	5	5
hotel-parking	4	4	4
hotel-pricerange	7	5	4
hotel-type	5	5	4
restaurant-pricerange	5	4	4
hotel-internet	3	3	3
Table 10: Statistics of domain transition in the test set of MultiWOZ 2.1. There are 140 dialogues with more than one domain transition that end with taxi domain. The cases where domain switches more than once and ends in taxi are shown in bold. The total number of dialogues with more than one domain transition is 175. We can view these as complicated dialogues.
Domain Transition	Count
First	Second	Third	Fourth
restaurant	train	-	-	87
attraction	train	-	-	80
hotel	-	-	-	71
train	attraction	-	-	71
train	hotel	-	-	70
restaurant	-	-	-	64
train	restaurant	-	-	62
hotel	train	-	-	57
taxi	-	-	-	51
attraction	restaurant	-	-	38
restaurant	attraction	taxi	-	35
restaurant	attraction	-	-	31
train	-	-	-	31
hotel	attraction	-	-	27
restaurant	hotel	-	-	27
restaurant	hotel	taxi	-	26
attraction	hotel	taxi	-	24
attraction	restaurant	taxi	-	23
hotel	restaurant	-	-	22
attraction	hotel	-	-	20
hotel	attraction	taxi	-	16
hotel	restaurant	taxi	-	13
attraction	-	-	-	12
attraction	restaurant	train	-	3
restaurant	hotel	train	-	3
hotel	train	restaurant	-	3
restaurant	train	hotel	-	3
restaurant	taxi	hotel	-	3
attraction	train	restaurant	-	2
train	attraction	restaurant	-	2
attraction	restaurant	hotel	-	2
hotel	train	attraction	-	2
attraction	taxi	hotel	-	1
hotel	taxi	-	-	1
train	hotel	restaurant	-	1
restaurant	taxi	-	-	1
restaurant	train	taxi	-	1
hotel	restaurant	train	-	1
hotel	taxi	train	-	1
taxi	attraction	-	-	1
restaurant	train	attraction	-	1
attraction	train	hotel	-	1
attraction	train	taxi	-	1
restaurant	attraction	train	-	1
hotel	taxi	attraction	-	1
train	hotel	attraction	-	1
restaurant	taxi	attraction	-	1
hotel	attraction	restaurant	taxi	1
attraction	hotel	train	-	1
taxi	restaurant	train	-	1
B Inference Time Complexity (ITC)
Table 11: Inference Time Complexity (ITC) of each model. We report the ITC in both the best case and the worst case for more precise comparison.  $J$  indicates the number of slots, and  $M$  indicates the number of values of a slot.
Model	Inference Time Complexity
Best	Worst
SUMBT	Ω(JM)	O(JM)
DS-DST	Ω(J)	O(JM)
DST-picklist	Ω(JM)	O(JM)
DST Reader	Ω(1)	O(J)
TRADE	Ω(J)	O(J)
COMER	Ω(1)	O(J)
NADST	Ω(1)	O(1)
ML-BST	Ω(J)	O(J)
SOM-DST(ours)	Ω(1)	O(J)
Inference Time Complexity (ITC) proposed by Ren et al. (2019) defines the efficiency of a DST model using  $J$ , the number of slots, and  $M$ , the number of values of a slot. Going a step further from their work, we report ITC of the models in the best case and the worst case for relatively more precise comparison.
Table 11 shows ITC of several models in their best and worst cases. Since our model generates values for only the slots on which UPDATE operation has to be performed, the best case complexity of our model is  $\Omega(1)$ , when there is no slot whose operation is UPDATE.
C Sample Outputs
Figure 3: The output of SOM-DST in a dialogue (dialogueidx MUL2499) in the test set of MultiWOZ 2.1. Parts changed from the previous dialogue state are shown in blue. To save space, we omit the slots with value NULL from the figure.
Figure 4: The output of SOM-DST in a dialogue (dialogueidx PMUL3748) in the test set of MultiWOZ 2.1. Parts changed from the previous dialogue state are shown in blue. To save space, we omit the slots with value NULL from the figure.