SlowGuess/NAACL_2021_OCR huggingface dataset:
We used PyTorch as a core framework for the three approaches. External packages we used include JointBERT $^5$  for IC/SL, edit-distance $^6$  for string similarity, and transformers $^7$  for the BERT-base $^8$  (for all the three approaches) and RoBERTa (for NLI). In addition, we used the softmax temperature of 0.1 to convert raw entity linking scores to probability in the probabilistic pipeline approach.
5.3 IC/SL and Entity Linking Results
We first consider the performance of individual components in the pipeline approaches. Table 2 shows the performance of JointBERT for intent classification and slot labelling. It can be seen that JointBERT performed better on the insurance domain for both IC and SL and this trend is consistent with the results of the original MultiDoGO paper.
For entity linking, we used several evaluation metrics, all of which were only computed when the intents were correctly classified. These include (1) Link accuracy: Given that the SL module detects the value of the correct slot type, link accuracy shows how likely the value is linked to the correct entity (including None). (2) None recall: The recall of None being predicted. This metric shows how often it can detect when entity mentions cannot be linked. It is also related to the ability of detecting closed type constraint violations. (3) Precision, Recall, F1: Considering all the turns in the test data, compare the predicted entities to the ground truth entities. (These metrics are affected by the performance of IC/SL. If the SL module incorrectly detects the slot type, this could cause low precision, recall, and F1 at the entity level here. In contrast, if the SL module does not detect the slot value, no text will be fed to the entity linker and the entity will not be predicted. This could cause low recall but would not affect the precision.)
$^{5}$ https://github.com/monologg/JointBERT  
 $^{6}$ https://pypi.org/project/  
edit-distance/  
 $^{7}$ https://huggingface.co/transformers/  
 $^{8}$ BERT-base makes our models have  $\sim 110$ M parameters.
Method (Threshold)	Insurance	Fast food
Link Accuracy	None Recall	F1	Precision	Recall	Link Accuracy	None Recall	F1	Precision	Recall
Exact match	93.97	85.82	87.47	85.45	89.58	76.10	81.03	70.74	70.37	71.12
Bijaccard	81.30	-	75.39	73.60	77.27	76.69	-	71.35	69.05	73.81
Levenshtein	77.58	-	71.95	70.25	73.75	71.87	-	66.30	64.18	68.56
NLI	76.50	-	70.95	69.27	72.73	71.68	-	66.96	65.15	68.87
NLI (0.8)	92.58	78.01	86.12	84.07	88.26	81.51	56.89	75.35	73.60	77.19
Average	78.05	-	72.39	70.67	74.19	76.79	-	71.81	69.70	74.04
Average (0.5)	94.74	80.85	88.19	86.15	90.32	84.07	62.06	77.77	75.96	79.66
Table 3: Entity linking results (in %). ' -' means that the entity linking method does not support 'open' entity linking. In other words, it cannot predict None for unlinkable entity mentions.
Table 3 shows the results of entity linking on two MultiDoGO domains. The simplest method, exact match, yielded acceptable results for the fast food domain and surprisingly good results for the insurance domain. This is because possible entities in the insurance domain (with the types car_brand, car_model, and car_year) usually have only one surface form. For example, we can only say "Honda" to refer to the "Honda" car brand entity. Meanwhile, the slot types of the fast food domain are much more flexible such as food_item and ingredient. A user may say only "meatball" or "meatballs" to refer to the "italian meatballs" entity in the bot schema. Besides, the difference between the two domains is partly because the IC/SL model worked better on the insurance domain and provided more accurate slot values to the entity linking step.
Because exact match is a very strict condition, it predicted None more often than other methods and got the highest None recall, while some other methods do not support open entity linking (including Bijaccard, Levenshtein, NLI, and Average) and got zero None recall. However, applying reasonable None thresholds to NLI and Average boosted up the results for all the metrics. The Average method with the threshold of 0.5 achieved the best link accuracy and F1 for both the insurance domain and the fast food domain. Overall, the results highlight that using a combination of methods results in better entity linking performance.
5.4 Violation Detection Results
This section discusses the overall constraint violation detection results with respect to the following metrics. (1) Turn correct: The proportion of the turns where the violation prediction is exactly correct for all constraints. (2) Turn IoU: The IoU score showing how much overlapping the predicted and the ground truth violations of a given turn are, on average. Let  $P$  and  $G$  be sets of predicted and
Figure 3: Examples of violation predictions of the three approaches. Note that the entity linking of the pipeline approaches here is Average (0.5).
ground truth violations of a given turn, respectively. IoU (Intersection over Union) of this turn equals  $\frac{|P\cap G|}{|P\cup G|}$ . (3) Conversation correct: The proportion of conversations where the violation predictions are correct for all the turns. (4) Precision, Recall, and F1: Consider each violation of a constraint as a positive instance, calculate precision, recall, and F1 of the violations being predicted.
As shown in Table 4, the deterministic pipeline
Method (Threshold)	Insurance	Fast food
Conver. correct	Turn correct	Turn IoU	F1	Preci-sion	Recall	Conver. correct	Turn correct	Turn IoU	F1	Preci-sion	Recall
Deterministic Pipeline Approach (DP)
Exact match	81.6	89.9	92.4	71.7	62.1	85.0	30.7	45.0	59.6	59.7	49.1	76.1
Bijaccard	74.9	85.6	88.4	39.2	70.6	27.1	39.4	52.2	63.0	51.5	69.8	40.8
Levenshtein	73.4	84.6	87.8	40.9	63.3	30.2	34.5	48.5	60.3	51.7	64.2	43.3
NLI	72.8	84.3	87.8	43.6	63.1	33.4	36.7	49.6	59.4	46.2	64.4	36.0
NLI (0.8)	80.5	89.6	91.9	70.1	62.6	79.6	36.7	48.3	61.9	58.2	54.4	62.4
Average	74.3	85.0	88.2	42.3	67.3	30.8	39.9	52.6	63.3	50.8	68.5	40.3
Average (0.5)	82.2	90.4	92.5	71.6	63.9	81.4	37.4	50.2	63.5	59.5	54.5	65.4
Probabilistic Pipeline Approach (PP)
Bijaccard	74.1	84.8	88.4	44.6	66.9	33.5	37.7	50.8	62.7	52.4	67.3	42.9
Levenshtein	73.7	84.6	88.0	44.3	63.8	33.9	31.9	46.2	58.4	51.2	62.0	43.5
NLI	70.7	83.1	86.8	44.0	58.7	35.2	34.3	47.0	58.3	49.0	62.3	40.3
NLI (0.8)	70.2	83.8	86.4	60.9	52.6	72.3	36.5	47.9	61.6	58.4	54.7	62.8
Average	73.7	84.7	88.2	45.1	64.4	34.6	35.0	48.7	60.8	52.8	64.0	45.0
Average (0.5)	75.4	85.8	89.3	52.5	57.8	48.1	38.2	50.8	63.8	59.0	55.6	63.0
End-to-End Approach (EE)
End-to-End BERT	83.9	92.1	93.4	75.1	76.2	74.1	33.3	52.0	62.4	57.4	60.0	55.1
Table 4: Constraint violation detection results (in %) of the three approaches. For the pipeline approaches, we compare using different entity linking methods and their impact on the constraint violation detection task.
approach (DP) with exact match as the entity linking method got the highest violation recall. This is because the exact match is good at detecting unlinkable slot values (see None recall in Table 3), so it got high recall concerning violation detection of closed type constraints. Conversely, entity linking methods which could not predict None (i.e., Bijaccard, Levenshtein, NLI, Average) got significantly lower violation recall and, hence, F1.
Furthermore, the difference between the two domains in Table 4 are more prominent than what we see for individual steps in Table 2-3. There are several reasons for this. First, the fast food domain has more custom slot types and more constraints. So, it is more difficult to predict violations of all the constraints correctly for each turn – resulting in lower conversation correct and turn correct. Second, for the pipeline approaches, the errors of individual steps of the fast food domain were higher than the errors of the insurance domain; therefore, the gap became larger when the errors were accumulated in the last step. An example in Figure 3(A) illustrates this case. The slot labelling part of Joint BERT identified “white top” and “pizza” as two separate food items. The entity linker, “Average (0.5)”, could not map “white top” to any of the defined entity. The system then understood that the user ordered an unknown food item and returned the closed_type_food_item violation which is incorrect. However, we did not see this particular error with the end-to-end approach.
Comparing the deterministic pipeline (DP) and
the probabilistic pipeline (PP) approaches, we can see that DP outperformed PP in most settings, especially in the insurance domain. We believe that when the entity linking module works accurately (as in the insurance domain), switching from DP to PP probably harms the overall performance since PP adds unnecessary uncertainty to the correct entity predictions. Conversely, when entity linking is a challenging step, PP with an appropriate softmax temperature could yield better results.
According to Table 4, the end-to-end approach (EE) clearly outperformed DP and PP in the insurance domain while being competitive to DP and PP in the fast food domain. This might be because the insurance domain has only one domain-specific (binary) constraint and three closed type (unary) constraints that are easier to learn from the training data. Meanwhile, the fast food domain has twelve binary and six unary constraints, respectively. Without access to the constraint statements, the existing training examples may not be sufficient to teach the end-to-end model all possible cases of the constraints. An example in Figure 3(B) shows that EE falsely returned the food_item-ingredient invalid violation in response to the input "Hai, I need BBQ chicken pizza with cheese" although this sentence in fact did not violate the constraint. This error might be because the model had not seen the combination of BBQ chicken pizza and cheese during training and it did not have access to the constraints defined in the bot schema.
Focusing on goal-oriented dialogue systems, we proposed a novel task - slot constraint violation detection - in NLU, together with constraint representation and three approaches to tackle this problem. While the pipeline approaches apply constraints as a post-processing step after IC/SL, the end-to-end approach attempts to model constraints inside the NLU. This sets the stage for future research and modeling of slot constraints and knowledge within NLU. In particular, there are several ways to enhance the end-to-end approach. For example, we could perform joint learning of IC, SL, and constraint violation detection to share the learned knowledge among tasks. Also, injecting logical constraints into BERT is an interesting direction. One way to do so is to translate constraints into violating and non-violating examples (by generating conversations with templates derived from existing training examples) and use them to train BERT together with other training examples. In addition, using constraints information, one can control the training data generation and the percentage of data with constraint violations depending on expected user behavior.
We would like to thank Jason Krone, Yi Zhang, Arshit Gupta, and anonymous reviewers for their helpful comments.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-35.
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling.
Zheng Chen and Heng Ji. 2011. Collaborative ranking: A case study on entity linking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 771-781, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Zheng Chen, Suzanne Tamang, Adam Lee, Xiang Li, Wen-Pin Lin, Matthew G Snover, Javier Artiles, Marissa Passantino, and Heng Ji. 2010. Cunybinder tac-kbp2010 entity linking and slot filling system description. In TAC.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-ConNLL), pages 708-716, Prague, Czech Republic. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 1625-1628.
Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and YunNung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 753–757, New Orleans, Louisiana. Association for Computational Linguistics.
Swapna Gottipati and Jing Jiang. 2011. Linking entities to a knowledge base with query expansion. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 804-813, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Arshit Gupta, John Hewitt, and Katrin Kirchhoff. 2019. Simple, fast, accurate intent classification and slot labeling for goal-oriented dialogue systems. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 46-55, Stockholm, Sweden. Association for Computational Linguistics.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2681-2690, Copenhagen, Denmark. Association for Computational Linguistics.
P. Haffner, G. Tur, and J. H. Wright. 2003. Optimizing svms for complex call classification. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03), volume 1, pages I-I.
Dilek Hakkani-Tur, Gokhan Tur, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional rnn-lstm. In *Interspeech*, pages 715-719.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579.
Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016. Leveraging sentence-level information with encoder LSTM for semantic slot filling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2077-2083, Austin, Texas. Association for Computational Linguistics.
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710.
E. Levin, R. Pieraccini, and W. Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on Speech and Audio Processing, 8(1):11-23.
Changliang Li, Liang Li, and Ji Qi. 2018. A self-attentive model with gate mechanism for spoken language understanding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3824–3833, Brussels, Belgium. Association for Computational Linguistics.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2060-2069, New Orleans, Louisiana. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468-1478, Melbourne,
Australia. Association for Computational Linguistics.
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal-oriented dialogues (MultiDoGO): Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4526-4536, Hong Kong, China. Association for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855.
W. Shen, J. Wang, and J. Han. 2015. Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engineering, 27(2):443-460.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Jason D. Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech and Language, 21(2):393 - 422.
Jing Xu, Liang Gan, Mian Cheng, and Quanyuan Wu. 2018. Unsupervised medical entity recognition and linking in chinese online medical text. Journal of healthcare engineering, 2018.
P. Xu and R. Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 78-83.
S. Young, M. Gašić, B. Thomson, and J. D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 154-167, Barcelona, Spain (Online). Association for Computational Linguistics.
Zhiwei Zhao and Youzheng Wu. 2016. Attention-based convolutional neural networks for sentence classification. In *Interspeech* 2016, pages 705-709.
Metric	Insurance	Fast food
Train
Number of dialogues	1,632	1,448
Number of user turns	13,821	13,038
Avg. turns per dialogue	8.47	9.00
# of non-violating dialogues	1,214	457
# of non-violating turns	11,800	6,303
% of non-violating dialogues	74.39	31.56
% of non-violating turns	85.38	48.34
Dev
Number of dialogues	233	214
Number of user turns	1,958	1,868
Avg. turns per dialogue	8.40	8.73
# of non-violating dialogues	172	79
# of non-violating turns	1,679	961
% of non-violating dialogues	73.82	36.92
% of non-violating turns	85.75	51.45
Test
Number of dialogues	467	414
Number of user turns	3,896	3,767
Avg. turns per dialogue	8.34	9.10
# of non-violating dialogues	358	145
# of non-violating turns	3,366	1,822
% of non-violating dialogues	76.66	35.02
% of non-violating turns	86.40	48.37
Table 5: Additional dataset statistics by data splits (training, development, and test).
All experiments was performed on one NVidia V100 GPU having 16 GB of VRAM.
C A Full Conversation Example
Table 6-7 show the violation detection results of a full conversation predicted by the three baseline approaches, together with the intermediate results from the deterministic pipeline approach (including intents, slots, entities, and dialogue states).
Turn	Input	Ground truth IC/SL/Entity linking	Pipeline IC/SL/Entity linking	Violations
0	hello	Intent: OpeningGreeting
Slots: {}
Entities: {}
DS: {}	Intent: OpeningGreeting
Slots: {}
Entities: {}
DS: {}	GT: []
DP: []
PP: []
EE: []
2	i would like to order 2 veg out pizza can you help me with that	Intent: OrderPizzaIntent
Slots: {quantity: [2], food_item: [veg out pizza]}
Entities: {quantity: [2], food_item: [pizza]}
DS: {orderpizzaintent: {quantity: [2], food_item: [pizza]}}	Intent: OrderPizzaIntent
Slots: {quantity: [2], food_item: [veg out pizza]}
Entities: {quantity: [2], food_item: [pizza]}
DS: {orderpizzaintent: {quantity: [2], food_item: [pizza]} }	GT: []
DP: []
PP: []
EE: []
4	my address is 1834 eden drive richmond va 23228 and my contact number is 804 913 4348	Intent: ContentOnly
Slots: {}
Entities: {}
DS: {orderpizzaintent: {quantity: [2], food_item: [pizza]}	Intent: OutOfDomain ×
Slots: {}
Entities: {}
DS: {orderpizzaintent: {quantity: [2], food_item: [pizza]} }	GT: []
DP: []
PP: []
EE: []
6	i would like high rise	Intent: OrderPizzaIntent
Slots: {food_item: [high rise]}
Entities: {food_item: [None]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]}	Intent: OrderPizzaIntent
Slots: {food_item: [high rise]}
Entities: {food_item: [None]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]} }	GT: [closed_food_item, food_item-orderpizza]
DP: [closed_food_item, food_item-orderpizza]
PP: [closed_food_item, food_item-orderpizza]
EE: [closed_food_item, food_item-orderpizza]
8	yes i want 2 wine	Intent: [Confirmation, OrderDrinkIntent]
Slots: {quantity: [2], drink_item: [wine]}
Entities: {quantity: [2], drink_item: [None]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]}, orderdrinkintent: {quantity: [2], drink_item: [None]}	Intent: [Confirmation, OrderDrinkIntent]
Slots: {quantity: [2], drink_item: [wine]}
Entities: {quantity: [2], drink_item: [None]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]}, orderdrinkintent: {quantity: [2], drink_item: [None]} }	GT: [closed_food_item, food_item-orderpizza, closed Drink item]
DP: [closed_food_item, food_item-orderpizza, closed Drink item]
PP: [closed_food_item, food_item-orderpizza, closed Drink item]
EE: [closed_food_item, food_item-orderpizza, food_item-orderdrink, closed Drink item] X
10	i would prefer red wine	Intent: OrderDrinkIntent
Slots: {drink_item: [red wine]}
Entities: {drink_item: [red wine]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]}, orderdrinkintent: {quantity: [2], drink_item: [red wine]}	Intent: OrderDrinkIntent
Slots: {food_item: [red wine]}
Entities: {food_item: [None]}
DS: {orderpizzaintent: {quantity: [2], food_item: [None]}, orderdrinkintent: {quantity: [2], drink_item: [None]}, food_item: [None]} X	GT: [closed_food_item, food_item-orderpizza]
DP: [closed_food_item, food_item-orderpizza, food_item-orderdrink, closed Drink item]
PP: [closed_food_item, food_item-orderpizza, food_item-orderdrink, closed Drink item] X
EE: [closed_food_item, food_item-orderpizza]
Table 6: A full conversation example for constraint violation detection (1/2). The results in the pipeline IC/SL/Entity linking column are obtained from JointBERT and Average (0.8) entity linking method. DS stands for dialogue states. GT, DP, PP, and EE are violation results of ground truth, the deterministic pipeline approach, the probabilistic pipeline approach, and the end-to-end approach, respectively.