SlowGuess/NAACL_2022_OCR huggingface dataset:
Identification Policy. The task-level policy queries the KB to retrieve candidate profiles (see Subsection 4.1), determines when to proceed to the next item, and decides when to terminate the identification process. The policy queries the KB with the NLU n-best results, and sends the retrieved profiles to the DST. Similarly to verification, the policy (re)prompts the user about an item until either
Table 2: Results for enrolment task: Precision (P), Recall (R), F1 score, and average number of turns (L) for exact match of the whole profile and each of its items (postcode, full name, and date of birth (DoB)).
Turns (Subsection 3.2)	Postcode	Name	DoB
P%	R%	F1%	P%	R%	F1%	P%	R%	F1%
en-GB	single(Qi), i=1,4,7	68.17	32.80	44.29	67.35	61.71	64.40	81.48	69.00	74.73
single(Qi), i=2,5,8	73.27	39.02	50.92	65.47	56.72	60.78	79.64	66.98	72.76
single(Qi), i=3,6,9	75.95	37.64	50.34	20.03	10.26	13.57	86.31	71.97	78.49
multi (Q1-9)	69.08	55.20	61.37	65.88	64.88	65.38	80.37	78.97	79.66
pHPL	single(Qi), i=1,4,7	95.95	58.26	72.50	74.11	62.98	68.10	93.69	76.04	83.95
single(Qi), i=2,5,8	97.37	79.96	87.81	73.62	62.08	67.36	93.33	77.30	84.56
single(Qi), i=3,6,9	97.53	85.33	91.03	21.95	6.68	10.24	93.80	81.27	87.08
multi (Q1-9)	95.51	91.91	93.68	71.86	69.26	70.54	92.92	90.31	91.59
fr-FR	single(Qi), i=1,4,7	80.76	51.59	62.96	45.06	42.86	43.93	91.21	73.42	81.36
single(Qi), i=2,5,8	82.48	65.02	72.72	41.44	39.72	40.56	92.91	74.61	82.76
single(Qi), i=3,6,9	83.09	65.07	72.98	2.64	1.85	2.18	92.02	76.08	83.29
multi (Q1-9)	77.62	72.09	74.75	44.21	44.00	44.10	90.81	86.81	88.76
Table 3: Results for single- vs multi-turn value extraction with cautious NLU: Precision (P), Recall (R), F1 score per item (postcode, full name, and date of birth).
the DST returns a well-defined score (Eq. 4) or the top-level policy reaches the limit for attempts (again, 3). The policy terminates after having exhausted all items, or when the anytime result of identification is an empty list and the KB cannot be queried by any upcoming item. Upon termination, the policy returns the ranked list of identified profiles.
4.5 Evaluating the EVI Tasks
Evaluating Enrolment. Suitable evaluation metrics come from the area of information extraction: precision (P), recall (R), and  $F1$  score, at the profile level or per item.[12]
Evaluating Verification. The relevant literature describes two basic metrics (El-Abed et al., 2012): False Rejection Rate (FRR) is the proportion of genuine users that the system incorrectly rejects as impostors; conversely, False Acceptance Rate (FAR) is the proportion of impostors that the system incorrectly accepts as genuine. Lower FRR indicates more usable systems, and lower FAR, more secure, e.g.  $\mathrm{FRR} = 1\%$  at  $\mathrm{FAR} = 1 / 10000$  means that  $1\%$  of genuine users will fail verification at the security level that falsely accepts 1 impostor per 10,000 impostor attempts. Equal Error Rate
11The expression is based on the  $L^p$  -norm,  $\| x\| _p\coloneqq \left(\sum_{i = 1}^{n}|x_i|^p\right)^{1 / p}$  , and is related to the generalised (aka power or Hölder) means (Bullen, 2013).
<sup>12</sup>Enrolment outputs (new profiles) are stored in the KB and fed into I&V downstream tasks (Fig. 1); evaluating interactions among tasks is outside the scope of this paper.
models	en-GB	pl-PL	fr-FR
nlu	V-model	EER%	FRR%	L	EER%	FRR%	L	EER%	FRR%	L
cautious	random	32.95	54.70	4.15 (2.85)	17.28	30.99	3.98 (2.67)	22.50	49.83	3.85 (2.38)
cautious	exact	28.22	56.42	4.15(2.78)	17.60	35.20	3.98 (2.59)	27.48	54.95	3.85 (2.30)
cautious	fuzzy	22.47	24.27	4.15 (3.09)	6.88	11.24	3.98 (2.76)	11.01	29.06	3.85 (2.57)
seeking	random	31.86	58.67	3.86 (2.59)	17.83	38.93	3.69 (2.37)	24.11	49.22	3.63 (2.30)
seeking	exact	30.89	61.77	3.86 (2.50)	21.15	42.29	3.69 (2.31)	25.87	51.73	3.63 (2.25)
seeking	fuzzy	11.27	21.06	3.86 (2.84)	4.27	10.56	3.69 (2.53)	9.11	18.73	3.63 (2.53)
Table 4: Results of verification task: Equal Error Rate (EER), False Rejection Rate (FRR) @FAR = 1/10,000, and average number of turns (L; in parentheses: with early termination @FAR = 1/10,000).
(EER) is the error rate when  $\mathrm{FAR} = \mathrm{FRR}$ ; it is a popular evaluation metric when a security level is not a priori specified. Finally, the Detection Error Trade-off (DET) graph plots FRR (y-axis) against FAR (x-axis) for varying values of the verification threshold  $(\theta)$  to visualise usability across a range of security levels (Martin et al., 1997).
Evaluating Identification. We rely on the identification rate at rank  $r$  (IR@r) (El-Abed et al., 2012): the proportion of identification transactions by pre-enrolled users in which the correct profile is among the top- $r$  retrieved by the system. It is equivalent to the familiar recall at rank metric from information retrieval (Manning et al., 2008).
This section evaluates benchmarks and empirically explores the unique challenges of each EVI task.
Experimental Setup. For all experiments, we deterministically simulate ground truths and user inputs from our EVI KB and dataset, respectively (see Subsections 3.1 and 3.2). The implementations of ASR, top-level policy, NLG, and TTS were set at the time of data collection and are common for all EVI tasks (see Subsection 4.1). Subsection 4.5 describes the evaluation metrics for each task.
5.1 Enrolment Experiments
We evaluate the enrolment policy with cautious or seeking NLU (see Subsection 4.1).
Results. Table 2 shows the impact of NLU on enrolment task accuracy (i.e. precision, recall, F1), for the whole profile and per item, and the average dialogue length. For whole profiles and almost all items, cautious NLU, which is more conservative and extracts fewer values, yields better accuracy than seeking NLU, which is more liberal and over-extracts values. Notably, extraction of French names
Figure 3: Detection Error Trade-off (DET) curves for the en-GB locale. A curve that is closer to the bottom of the plot corresponds to better verification performance.
and English postcodes (alphanumeric) was less accurate than for other locales (digit-only postcodes).
Further Analysis. Table 3 shows per item the accuracy (i.e. precision, recall, F1) of single- and multi-turn value extraction with the cautious model. Consistently, recall with multi-turn extraction is higher than single-turn recall of any individual turn. Conversely, individual single-turns yield the highest precisions. Across locales, the relevant precisions of turns is retained for postcodes  $(Q_{3} > Q_{2} > Q_{1})$  and names  $(Q_{4} > Q_{5} > Q_{6})$  (cf. Section 3.2). In particular, extraction of name spellings  $(Q_{6})$  is distinctly poor; this barely affects multi-turn performance, because, on average, the system collects names before  $Q_{6}$  (Table 2).
5.2 Verification Experiments
We evaluate the verification policy with cautious or seeking NLU and random, exact, or fuzzy verification (Subsection 4.3) on the EVI dataset and KB (Section 3), from which we sample genuine and impostor profiles at a  $1:1$  ratio.
Results. Table 4 shows the impact of NLU and verification models on the equal error rate (EER), the FRR at the FAR = 1/10 000 security level and length. Consistently, seeking NLU with
fuzzy verification yields the best EER and FRR. Interestingly, exact verification fails to improve reliably over the random baseline. Finally, early termination shortens verification length by  $25 - 30\%$ .
Further Analysis. Figure 3 shows the DET curves for the en-GB locale and all models. Exact verification produces single points on the y-axis, which we linearly interpolate to produce its DET curve. Again, seeking NLU with fuzzy verification yields the best usability-security trade-off (lowest-lying curve) for the whole range of security levels in the graph. The same holds for the DET curves of the pl-PL and fr-FR (shown in Appendix E).
5.3 Identification Experiments
We evaluate the identification policy with cautious or seeking NLU (Subsection 4.1), and no (none), exact, fuzzy, or oracle (upper bound) identification (Subsection 4.4). We vary the  $\alpha$  parameter of the infinity-one p-norm (Eq. 7).
Results. Table 5 shows the impact of NLU and identification models on identification rate at rank 1 and identification length. Without an explicit identification model (none) the agent cannot differentiate among multiple retrieved profiles and accuracy is very low. Consistently, seeking NLU, fuzzy models, and  $\alpha = 0.5$  perform better than cautious NLU, exact matching, and  $\alpha = 1$  (i.e. the standard fuzzy operators), respectively. These effects are orthogonal: seeking NLU with fuzzy model and  $\alpha = 0.5$  produces the best accuracy, almost on par with the oracle.
Further Analysis. Most identification errors  $(>98\%)$  were caused by low recall: the correct target profile was not included in those returned by querying the KB with the NLU results, which is reminiscent of the unlinkable entity (NIL) problem from entity linking (Ling et al., 2015; Hoffart et al., 2014; McNamee and Dang, 2009). Table 6 shows the upper bounds using a KB oracle (Subsection 4.1), and corroborates the results of Table 5. The best combination (seeking NLU, fuzzy model and  $\alpha = 0.5$ ) can achieve almost perfect performance as an upper bound.
5.4 Directions for Further Research
Our findings highlight the most promising directions for further improvements. In particular, for enrolment: high-precision NLU and multi-turn belief tracking; for verification: high-recall NLU and fuzzy matching; and for identification:
models	en-GB	pl-PL	fr-FR
nlu	I-model	IR@1	L	IR@1	L	IR@1	L
cautious	none	9.90	3.64	19.74	3.86	14.95	3.62
seeking	none	10.04	3.54	19.89	3.71	15.09	3.46
cautious	exact(α=1)	50.22	3.64	65.90	3.86	48.50	3.62
cautious	fuzzy(α=1)	64.88	3.64	89.15	3.86	71.00	3.62
seeking	exact(α=1)	46.75	3.54	61.93	3.71	52.40	3.46
seeking	fuzzy(α=1)	66.18	3.54	93.82	3.71	79.73	3.46
cautious	exact(α=0.5)	66.11	3.64	94.22	3.86	79.31	3.62
cautious	fuzzy(α=0.5)	66.33	3.64	94.32	3.86	78.97	3.62
seeking	exact(α=0.5)	67.27	3.54	94.88	3.71	80.35	3.46
seeking	fuzzy(α=0.5)	67.77	3.54	95.13	3.71	80.83	3.46
cautious	oracle	66.55	2.12	94.37	1.56	80.92	1.75
seeking	oracle	67.99	2.09	95.38	1.52	81.02	1.73
Table 5: Results of identification task: Identification Rate at rank 1 (IR@1) and average dialogue length (L).
models	en-GB	pl-PL	fr-FR
nlu	I-model	IR@1	L	IR@1	L	IR@1	L
seeking	none	15.53	3.86	20.54	3.69	18.46	3.63
seeking	exact(α=1)	38.22	3.86	57.71	3.69	48.27	3.63
seeking	fuzzy(α=1)	81.86	3.86	95.63	3.69	90.18	3.63
seeking	exact(α=0.5)	96.60	3.86	97.79	3.69	97.63	3.63
seeking	fuzzy(α=0.5)	98.19	3.86	98.74	3.69	98.81	3.63
seeking	oracle	100.00	1.00	100.00	1.00	100.00	1.00
Table 6: Identification task with a KB oracle.
high-recall NLU, fuzzy retrieval, and boosting the recall of querying the KB. All tasks can benefit from better multilingual NLU, and our dataset includes audios to encourage improvements in ASR.
Authentication Tasks. Our EVI tasks seek to automate the process of knowledge-based authentication (Braz and Robert, 2006; O'Gorman, 2003) in a voice communication context (O'Gorman et al., 2006a,b; O'gorman et al., 2005) using task-oriented spoken dialogue systems. We define and evaluate the tasks analogously to automated systems for biometric authentication (signatures, Yeung et al., 2004; fingerprints, Maio et al., 2002; faces, Phillips et al., 2003; irides, Phillips et al., 2008; and voice, Doddington et al., 2000).
Dialogues, NLP, and Logic. Our EVI benchmarks focus on speech recognition and spoken language understanding of names (Kaplan, 2020; Pappu and Rudnicky, 2014), dates (Price et al., 2021), and spellings (Vertanen and Kristensson, 2012; Filisko and Seneff, 2004; Chung et al., 2003). Furthermore, enrolment is a particular case of the slot-filling dialogue task (Young, 2002; Bellegarda, 2014); and identification is related to information retrieval and shares challenges with entity linking (Ling et al., 2015; Hoffart et al., 2014; McNamee and
Dang, 2009). We extend fuzzy logic methods from information retrieval (Radecki, 1979; Zadrożny and Nowacka, 2009; Salton et al., 1983) and from multi-modal verification (Lau et al., 2004; Conti et al., 2007; Azzini et al., 2007) to the context of spoken dialogues.
Dialogue Datasets. Research in dialogue systems is driven by competitions (Kim et al., 2019; Gunasekara et al., 2020) and challenge datasets, which may be human-to-human (Schrading et al., 2015; Lowe et al., 2015; Ritter et al., 2010), machine-to-machine (Shah et al., 2018), or human-to-machine (H2M) conversations; about single (Coope et al., 2020; Wen et al., 2017; Hemphill et al., 1990) or multiple domains (Rastogi et al., 2020; Zhu et al., 2020; Zang et al., 2020; Budzianowski et al., 2018; El Asri et al., 2017); in one or several languages (Xu et al., 2020; Li et al., 2021); and with written or spoken data (Lugosch et al., 2019; Li et al., 2018; Hemphill et al., 1990). Our EVI dataset is a spoken-language, multi-lingual, single-domain, human-to-machine challenge dataset for multiple tasks, which were not covered by any dialogue dataset from prior work.
We introduced novel spoken-dialogue tasks (knowledge-based enrolment, verification, and identification), the EVI multi-lingual spoken-dialogue dataset with 5,506 dialogues, and benchmark models, evaluations, and upper-performance bounds that leave ample margins for future improvements.
Limitations. During data collection, our policy (fixed-length with reprompts for all items) might have caused artefacts in speaker behaviour (e.g. frustration, chuckling, simplification for later items). Additionally, speaker behaviour of crowdsourced speakers who impersonate a fake profile will be qualitatively different to presenting one's own personal information (e.g. a young female speaker might be asked to impersonate an older male profile); however, ethical and privacy concerns preclude the publication of a dataset with real data. Finally, our current evaluation considers each downstream task in isolation, although in practice they form a sequence (enrolment, identification, and then verification) that may propagate errors.
Future Work. We invite the community to work on the novel EVI tasks and challenge dataset, which pose a variety of unresolved technical challenges: speech recognition, multi-turn spoken language
understanding, fuzzy matching and retrieval, etc.
We are grateful to our colleagues at PolyAI for our many fruitful discussions. We also thank the anonymous reviewers for their helpful suggestions.
PolyAI is ISO27k-certified and fully GDPR-compliant. Before data collection, we informed the crowd-sourced human workers that their voluntary participation will allow us to collect, store, publish, and use their fully-anonymous data for research purposes. During data collection, we did not ask workers for their own personal information (e.g. name, postcode); instead, we provided fictional (but realistic looking) profiles for them to impersonate. We instructed workers on how to hide their caller id, we did not store any inbound phone numbers, and we use fully anonymised identifiers in our dataset. Finally, we offered a fair compensation (around the average hourly wage in the UK, pro-rata) to all workers from all locales.
John Amein. 2020. Hidden risks of consumer-grade biometrics. Biometric Technology Today, 2020(10):5-8.
Antonia Azzini, Stefania Marrara, Roberto Sassi, and Fabio Scotti. 2007. A fuzzy approach to multimodal biometric authentication. In Proceedings of KES.
Jerome R Bellegarda. 2014. Spoken language understanding for natural interaction: The siri experience. Natural interaction with robots, knowbots and smartphones, pages 3-14.
Christina Braz and Jean-Marc Robert. 2006. Security and usability: the case of the user authentication methods. In Proceedings of l'Interaction Homme-Machine.
Susan E Brennan and Herbert H Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of experimental psychology: Learning, memory, and cognition, 22(6):1482.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP.
Peter S Bullen. 2013. Handbook of means and their inequalities, volume 560. Springer Science & Business Media.
Grace Chung, Stephanie Seneff, and Chao Wang. 2003. Automatic acquisition of names using speak and spell mode in spoken dialogue systems. In Proceedings of NAACL-HLT.
Vincenzo Conti, Giovanni Milici, Patrizia Ribino, Filippo Sorbello, and Salvatore Vitabile. 2007. Fuzzy fusion in multimodal biometric systems. In Proceedings of KES.
Samuel Coope, Tyler Farghly, Daniela Gerz, Ivan Vulic, and Matthew Henderson. 2020. Span-convert: Few-shot span extraction for dialog with pretrained conversational representations. In Proceedings of ACL.
Rachel Coulston, Sharon Oviatt, and Courtney Darves. 2002. Amplitude convergence in children's conversational speech with animated personas. In Proceedings of ICSLP.
George R Doddington, Mark A Przybocki, Alvin F Martin, and Douglas A Reynolds. 2000. The nist speaker recognition evaluation—overview, methodology, systems, results, perspective. Speech communication, 31(2-3):225–254.
Mohamad El-Abed, Romain Giot, Baptiste Hemery, and Christophe Rosenberger. 2012. Evaluation of biometric systems: A study of users' acceptance and satisfaction. International Journal of Biometrics, 4(3):265-290.
Layla El Asri, Hannes Schulz, Shikhar Kr Sarma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of SIGDIAL.
Edward Filisko and Stephanie Seneff. 2004. Error detection and recovery in spoken dialogue systems. In Proceedings of HLT-NAACL Workshop on Spoken Language Understanding for Conversational Systems and Higher Level Linguistic Information for Speech Processing.
Howard Giles, Nikolas Coupland, and IUSTINE Coupland. 1991. 1. Accommodation theory: Communication, context, and. Contexts of accommodation: Developments in applied sociolinguistics, 1.
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, et al. 2020. Overview of the ninth dialog system technology challenge: Dstc9. arXiv preprint arXiv:2011.06486.
Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The atis spoken language systems pilot corpus. In Proceedings of the Workshop on Speech and Natural Language. HLT '90.
Johannes Hoffart, Yasemin Altun, and Gerhard Weikum. 2014. Discovering emerging entities with ambiguous names. In Proceedings of TheWebConf.
Michal Hrabí. 2020. Call centres: going voice-first in the post-covid world. Biometric Technology Today, 2020(8):10-12.
Anil K Jain, Arun Ross, and Salil Prabhakar. 2004. An introduction to biometric recognition. IEEE Transactions on circuits and systems for video technology, 14(1):4-20.
Micaela Kaplan. 2020. May i ask who's calling? named entity recognition on call center transcripts for privacy law compliance. In Proceedings of the EMNLP Workshop on Noisy User-generated Text (WNUT).
Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al. 2019. The eighth dialog system technology challenge. arXiv preprint arXiv:1911.06394.
Chun Wai Lau, Bin Ma, Helen Mei-Ling Meng, Yiu-Sang Moon, and Yeung Yam. 2004. Fuzzy logic decision fusion in a multimodal biometric system. In Proceedings of ICSLP.
Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. 2018. Spoken squad: A study of mitigating the impact of speech recognition errors on listening comprehension. arXiv preprint arXiv:1804.00320.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of EACL.
Xiao Ling, Sameer Singh, and Daniel S Weld. 2015. Design challenges for entity linking. TACL, 3:315-328.
Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of SIGDIAL.
Loren Lusosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. arXiv preprint arXiv:1904.03670.
Dario Maio, Davide Maltoni, Raffaele Cappelli, James L. Wayman, and Anil K. Jain. 2002. Fvc2000: Fingerprint verification competition. IEEE transactions on pattern analysis and machine intelligence, 24(3):402-412.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval, Chapter 8. Cambridge University Press.
Alvin Martin, George Doddington, Terri Kamm, Mark Ordowski, and Mark Przybocki. 1997. The det curve in assessment of detection task performance. Technical report, NIST.
Paul McNamee and Hoa Trang Dang. 2009. Overview of the tac 2009 knowledge base population track. In Proceedings of TAC.
Bob Morgen. 2012. Voice biometrics for customer authentication. Biometric Technology Today, 2012(2):8-11.
Kate G Niederhoffer and James W Pennebaker. 2002. Linguistic style matching in social interaction. Journal of Language and Social Psychology, 21(4):337-360.
Lawrence O'Gorman. 2003. Comparing passwords, tokens, and biometrics for user authentication. volume 91, pages 2021-2040. IEEE.
Lawrence O'gorman, Amit Bagga, and Jon Bentley. 2005. Query-directed passwords. Computers & Security, 24(7):546-560.
L O'Gorman, L Brotman, and M Sammon. 2006a. Comparing authentication protocols for securely accessing systems by voice. In Proceedings of ICSKM.
Lawrence O'Gorman, Lynne Brotman, and Michael Sammon. 2006b. How to speak an authentication secret securely from an eavesdropper. In International Workshop on Security Protocols. Springer.
Aasish Pappu and Alexander Rudnicky. 2014. Knowledge acquisition strategies for goal-oriented dialog systems. In Proceedings of SIGDIAL.
Jennifer S Pardo. 2006. On phonetic convergence during conversational interaction. The Journal of the Acoustical Society of America, 119(4):2382-2393.
John Petersen. 2019. The complexity of consent and privacy in biometrics-worldwide. Biometric Technology Today, 2019(8):5-7.
P Jonathon Phillips, Kevin W Bowyer, Patrick J Flynn, Xiaomei Liu, and W Todd Scruggs. 2008. The iris challenge evaluation 2005. In Proceedings of the International Conference on Biometrics. IEEE.
P Jonathon Phillips, Patrick Grother, Ross Micheals, Duane M Blackburn, Elham Tabassi, and Mike Bone. 2003. Face recognition vendor test 2002. In Proceedings of the International SOI Conference. IEEE.
Martin J Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(2):169-190.
Ryan Price, Mahnoosh Mehrabani, Narendra Gupta, Yeon-Jun Kim, Shahab Jalalvand, Minhua Chen, Yanjie Zhao, and Srinivas Bangalore. 2021. A hybrid approach to scalable and robust spoken language understanding in enterprise virtual agents. In Proceedings of NAACL-HLT.
Tadeusz Radecki. 1979. Fuzzy set theoretical approach to document retrieval. Information Processing & Management, 15(5):247-259.