SlowGuess/NAACL_2022_OCR huggingface dataset:
Georgios P. Spithourakis, Ivan Vulic,
Michał Lis, Iñigo Casanueva, and Paweł Budzianowski
PolyAI Limited
{georgios, ivan, michal, inigo, pawel}@poly.ai
Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date of birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.
Computer systems need to be able to identify and verify their users before granting access to personalised services and confidential information (Braz and Robert, 2006; O'Gorman, 2003). In particular, identification (I) is the process of specifying the identity of a person, i.e. answer the question: "who are you?". On the other hand, verification (V) (aka authentication) is the process of confirming the assertion about a claimed identity, i.e. answer "are you who you claim you are?" (Jain et al., 2004). In both processes, the system compares information given by the user with information held by the system; thus they presume enrolment (E), that is, the process of registering the identity information of a new user into the system (Jain et al., 2004).
Task-oriented dialogue systems that offer personalised and privacy-focused services (e.g. set up utilities, track a parcel, or access a bank account) should be able to enrol, identify, and verify new and recurring users, without interrupting their natural conversational interface. Different types of authentication factors may be used (Smith, 2001; O'Gorman, 2003): i) knowledge-based ("what you know"), rely on a secret password or personal information, e.g.
Figure 1: Knowledge-based EVI for task-oriented spoken dialogue systems: enrolment (E) creates a new user profile to store in a KB; identification (I) retrieves a pre-enrolled profile for a user; and verification (V) asserts whether the user matches a claimed profile.
full name, date of birth, mother's maiden name, etc.; ii) possession-based ("what you have"), rely on possession of a physical token, e.g. a smart card, a metal key, etc.; and iii) inheritance-based ("who you are"), typically rely on biometric properties, e.g., a voiceprint, fingerprint, eye scan, or signature (Variani et al., 2014). Most businesses use knowledge-based authentication in their call centres to identify customers over the phone (Hrabí, 2020; Amein, 2020; Morgen, 2012; Petersen, 2019). As conversational AI is increasingly being used to automate call centres, we seek to enable task-oriented spoken dialogue systems with EVI functionalities.
The core contributions of this paper are:
1. We motivate and formalise knowledge-based enrolment, verification, and identification as novel tasks for task-oriented spoken dialogue systems (Section 2).
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1647 - 1659
2. We collect and publish a novel conversational dataset with 5,506 dialogues that can be used to develop and evaluate EVI-oriented spoken dialogue systems in 3 languages (British English, Polish, and French; Section 3). The multilingual aspect of the dataset allows us to also study language-specific variations in data and performance, reaching beyond monolingual, English-only setups.
3. We define baseline models and suitable evaluation protocols (Section 4) for the new tasks. Finally, we evaluate benchmarks on the new dataset, explore the unique challenges of these tasks, and set directions for future research (Section 5).
The code and dataset is available online at: https://github.com/PolyAI-LDN/evi-paper.
2 The EVI Dialogue Tasks
Preliminaries. For all tasks, we assume that the dialogue system can interact with a Knowledge Base (KB) of stored profiles,  $P_{KB} = \{p_1, p_2, \ldots\}$ . Each profile,  $p$ , is a structured record of a real-world entity (e.g. a user, product, etc.) that comprises one or more items, i.e. key-value pairs (e.g. postcode, name, date of birth, etc.). The user and system take alternate turns,  $t$ , that make up a multi-turn dialogue,  $T_{dialogue} = \{t_{1,system}, t_{1,user}, t_{2,system}, t_{2,user}, \ldots\}$ .
Enrolment Task. The goal of enrolment is to create and store a profile that represents the identity of a new user and that can be used to identify or verify the same user in the future. For dialogue-based enrolment, the system must be able to extract all required item key-value pairs from the dialogue to construct a new profile to store in the KB (cf. Fig. 1):
$$
p _ {\text {n e w}} = \operatorname {e n r o l} \left(T _ {\text {d i a g o u g e}}\right) \tag {1}
$$
Verification Task. The goal of verification is to decide whether a user who claims an identity is genuine or an impostor. For dialogue-based, knowledge-based verification, the system must be able to compare information stored in the KB about the claimed identity with information provided by the user in the dialogue to produce a verification score that quantifies the degree of the match (cf. Fig.1):
$$
s _ {\text {p r o f i l e}} = \operatorname {v e r i f y} \left(p _ {\text {c l a i m e d}}, T _ {\text {d i a l o g u e}}\right) \in [ 0, 1 ], \tag {2}
$$
where  $s = 1$  signifies a genuine verification attempt, and  $s = 0$  denotes an impostor verification attempt. The system designer can apply a threshold,  $\theta$ , to
obtain a crisp verification outcome and control the system's trade-off between security and usability (see later Subsections 4.3 and 4.5).
Identification Task. The goal of identification is to determine the identity of an unknown user from a KB of pre-enrolled user profiles. For dialogue-based, knowledge-base identification, the system must be able to query the KB with the information provided by the user in the dialogue to retrieve a ranked list of the best matching profiles (cf. Fig. 1):
$$
p _ {1}, p _ {2}, \dots = \text {i d e n t i f y} \left(P _ {K B}, T _ {\text {d i a l o g u e}}\right) \tag {3}
$$
The list might be empty if no qualifying profiles (i.e. above a score threshold) could be retrieved.
3 A Multilingual Spoken Dialogue Dataset
We set out to build a novel, first of its kind, human-to-machine conversational dataset that can be used to develop and evaluate task-oriented spoken dialogue systems that support the functionality of the knowledge-based EVI tasks. The dataset is multilingual and covers 3 locales: British English (en-GB), French (fr-FR), and Polish (pl-PL).<sup>1</sup>
3.1 Generating the Profiles Knowledge Base
For each locale, we populate a KB to be shared across EVI tasks. We randomly generated locale-dependent profiles using the faker tool. Each profile in the KB consists of its generated item key-value pairs for postcode, full name, and date of birth (cf. Fig. 1). These three different slots are popular in industrial authentication procedures. Because in the real world people might share the same name, postcode, or date of birth by coincidence, we allow duplicate values in our generated data, e.g. for each locale our KB contains 10,000 unique profiles, but only 2,000 unique postcodes. Table 1 shows the size of the generated KB.
3.2 Collecting the Dialogue Data
We developed a spoken dialogue system to collect the postcode, full name, and date of birth of a user over the phone. The system operates under a deterministic policy with static retries for each collection step. We use the same sequence of dialogue acts
<sup>1</sup>The choice of these languages was motivated by the popularity, the phonetic richness and a large enough base of high-quality crowdworkers.
$^{2}$ https://faker.readthedocs.io/; it is a python package that can generate fake but reasonable data (names, addresses, phone numbers, etc.) for bootstrapping databases.
for all EVI tasks, and vary the scripted prompts (see Subsection 3.3) to elicit more diverse responses:
Q1: What is your postcode?
Q2: Please tell me your postcode.
Q3: I heard [A B 1]. Please tell me your postcode.
Q4: What is your full name?
Q5: Please tell me your first and last name.
Q6: Please spell your full name.
Q7: What is your date of birth?
Q8: Please tell me your date of birth.
Q9: I heard [the 1st of January]. Please tell me your date of birth.
For other locales, see Appendix A. For each locale, we enlisted cohorts of speakers on the Prolific Academic (www.prolific.co) crowdsourcing platform. We displayed a random profile from the KB for each speaker to impersonate, e.g.:
Postcode:AB12CD
Kod Pocztowy: 12-345
Full Name: John Smith
Imie Nazwisko: Anna Krupa
Date of Birth: 4/7/1989
Data urodzenia: 1/1/2000
Then, we directed speakers to call a phone number to interact with our spoken dialogue system. To ensure quality, the crowdsourced speakers had to complete all turns of the static policy to receive their payment code.<sup>3</sup> Additionally, we filtered out all dialogues for which text-to-speech detected silence for all turns of a single item or for more than half of the turns of the dialogue.
For each turn, the EVI conversational dataset contains: the unique identifier of the impersonated profile from the KB; a unique speaker identifier; the raw audio data; the n-best list of transcriptions (see Appendix B); and any variation in the prompts (see Subsection 3.3). Table 1 shows the size of our dialogue dataset for all locales, which contains 5,506 dialogues in total.
3.3 Diversifying Speaker Behaviours
To elicit mode diverse behaviours from the speakers, and thus increase the generality and richness of our dataset, we exploited two psychological phenomena: priming and entrainment.
Priming is the psychological effect wherein exposure to a stimulus (prime) unconsciously influences the response to a later stimulus (target). Priming also affects linguistic decision making, e.g. exposure to a lexical item or syntactic structure reinforces reuse of the same pattern in the future (Reitter
et al., 2006, 2010). Likewise, entrainment is the phenomenon wherein conversational interlocutors adopt each other's linguistic patterns. Entrainment can be observed at multiple levels, e.g. lexical (Brennan and Clark, 1996), syntactic (Reitter and Moore, 2007), stylistic (Niederhoffer and Pennebaker, 2002), phonetic (Pardo, 2006), and prosodic (Coulston et al., 2002). The Interactive Alignment Model (Pickering and Garrod, 2004) proposes that conversational interlocutors automatically prime each other at multiple levels, causing their speech to converge.
Diversifying Spoken Dates. Our primes to diversify the speakers' lexical choice for dates were the formats that we used to lexicalise and display the dates of birth to the crowdsourced speakers. We used either of two formats at equal proportions:
(a) month=name: 1 January|stycznia|janvier 2000
(b) month=number: 1/1/2000
The Sankey diagram<sup>5</sup> in Figure 2 (top) shows that  $92\%$  of English speakers primed with the month  $\equiv$  name format echoed this pattern in  $Q_{7}$ , and only  $10\%$  of those switched to say the month's number in follow-up turns (similar results for pl-PL and fr-FR; see Appendix C for their Sankey diagrams). On the other hand, only  $54\%$  of English speakers (cf.  $26\%$  for pl-PL,  $36\%$  for fr-FR; Appendix C) primed with the month  $\equiv$  number format echoed that pattern in  $Q_{7}$ , and  $77\%$  of those switched to say the month's name later. Overall, the month  $\equiv$  name format (more lexical) had a stronger priming effect than the month  $\equiv$  number format (more symbolic), and speakers say the month's name (more verbose) increasingly after reprompts ( $Q_{8}$  and  $Q_{9}$ ).
Diversifying Spoken Spellings. Our primes to diversify the speakers' spelling choices were the agent reprompts in the  $Q_{3}$  that read back partial spellings of postcodes to the speaker. We used either of two strategies at equal proportion:
(a) spell=naive: A B one two CD
(b) spell=nato:6 Alfa Bravo one two Charlie Delta
4 Alternatively, Communication Accommodation Theory (Giles et al., 1991) proposes that more strategic decisions drive convergence (or divergence).
5Sankey diagrams visualise the flow or route of communication (or other quantity) within a system to help locate the most important contributions to a flow. The width of the links between nodes is proportional to the flow rate between them.
3The workers were not aware that the system was scripted, yielding the natural behaviour of irritated customers.
counts (unique)	Locale
en-GB	pl-PL	fr-FR
KB	#profiles	10,000	10,000	10,000
#postcodes	2,000	2,000	2,000
#names(first)	364	153	216
#names(last)	500	3,455	400
#names(full)	9,412	9,923	9,433
#DoBs	8,884	8,862	8,862
Dialogues	#dialogues	1,407	1,991	2,108
#turns	12,663	17,919	18,972
#speakers	1,081	803	521
#profiles	886	961	1,464
Table 1: Size of the created EVI Knowledge Bases and the collected Conversational Dataset.
These strategies acted as primes that entrained the speaker concerning their spelling strategy.
Figure 2 (bottom) shows that only  $1\%$  of en-GB speakers spontaneously used NATO spelling before/without encountering the spell=nato strategy in  $\mathrm{Q}_3$ . Conversely, using the spell=nato strategy entrained  $52\%$  of speakers to adopt that strategy in their response to  $\mathrm{Q}_3$ . Entrainment weakens over time: only  $28\%$  of entrained speakers remained entrained by  $\mathrm{Q}_6$ . Postcodes do not contain letters in the pl-PL and fr-FR locales, so both spelling strategies are equivalent. Only  $0.5\%$  of pl-PL and  $0.1\%$  of fr-FR speakers spontaneously used complex spelling strategies (listed in Appendix D).
In conclusion, we validated that priming and entrainment are effective tools to subtly guide speaker behaviour towards desired patterns. It is by varying those primes that we could increase the variability of speaker behaviours in our dataset.
4 EVI-oriented Spoken Dialogue Systems
This section presents the components of task-oriented spoken dialogue systems for EVI tasks and provides benchmark implementations for the upcoming experiments (see Sections 5.1, 5.2, and 5.3)
4.1 Components of EVI Dialogue Systems
Automatic Speech Recognition (ASR). When collecting the EVI dataset, we used Google's locale-specific speech-to-text $^{7}$  in streaming mode to derive n-best transcriptions and to implement quality control (see Subsection 3.2). Consequently, this is the ASR used in all experiments. The length
Figure 2: Sankey diagrams that visualise priming and entrainment of speaker behaviour for dates (top) and spelling (bottom) for the British English locale. Transitions in the direction of priming in red; against, in blue.
of the n-best lists was on average 4.85, 2.60, and 8.65 for English, Polish, and French, respectively (see Appendix B) and was capped at a maximum of 20 items.
Natural Language Understanding (NLU). For each item, we use an appropriate resource to extract values from the whole ASR n-best list into an NLU results n-best list. In our experiments, we first preprocess to normalise numbers ('one'  $\rightarrow$  '1') and letter spellings ('Bravol[B for B.*]  $\rightarrow$  'B'), and then extract values for postcodes using locale-dependent regular expressions ('A(A)9(A) 9AA' for en-GB; '99999' for pl-PL and fr-FR); for names, the lists of names from the US Census and other sources (Remy, 2021); and for dates, the dateparser package. Using these resources, we define two NLU models for value extraction: the cautious model requires whole-string match, whereas the seeking model searches for (potentially overlapping) substring matches.
Top-Level Policy. All EVI tasks share a common sequence of dialogue acts (DAs): the agent asks (request DA) the user to input the value (inform DA) of each profile item successively, with a limited number of re-prompts per item. In the experiments, the order of items is: postcode,
<sup>6</sup>The NATO phonetic alphabet substitutes a word for each letter to be easily understood in voice communications; https://www.nato.int/cps/en/natohq/declassified_136216.htm
<sup>7</sup>https://cloud.google.com/speech-to-text
<sup>8</sup>https://www.census.gov/topics/population/genealogy/data/1990_census/1990_census_namefiles.html
$^{9}$ https://dateparser.readthedocs.io/ it is a python package that can parse localised dates in any string format
full name, and date of birth, with up to 3 attempts per item (fixed at the time of dataset collection; see Subsection 3.2).
Task-Level Dialogue Management. Each of the three tasks requires task-specific dialogue state tracking (DST) and dialogue policy. The DST model tracks and updates the system's state and belief about the values of items and the candidate profiles, whereas dialogue policy selects the following system action (e.g. re-prompt user, proceed to next item, terminate task) and interacts with the profiles KB. We define the task-specific DST models and policies in more detail in Subsections 4.2, 4.3, and 4.4.
Integration with the Profiles KB. For enrolment, the system needs write access to the KB to store the extracted profile; for identification, the system needs read access to the KB to retrieve candidate profiles via a dynamic sequence of queries; and for verification, the claimed profile in the KB is previously made available from an upstream identification process (cf. Fig. 1). In the experiments, we do not explicitly model KB integration for enrolment (write-only access) and verification (downstream of identification); for identification, we model a read-only KB integration that supports querying by postcode (exact match) and an oracle that always includes the postcode of the correct profile in the query, regardless of the NLU results.
Natural Language Generation (NLG). When collecting the dataset, we used scripted prompts (Subsection 3.2) translated for each locale (Appendix A).
Text-to-Speech (TTS). We used Google's  $^{10}$  locale-specific TTS when collecting the EVI dataset.
4.2 Enrolment Models and Policies
Enrolment DST and Model. We track the value of each item, which is initially undefined. After each user input for an item, we may use the NLU n-best results to update its value. When the enrolment policy terminates, the enrolment model straightforwardly builds the new profile from the tracked items. In the experiments, we update an item's value with its latest top-1 result of the NLU (if not empty).
Enrolment Policy. The task-level policy determines when to proceed to the next item, and decides when to terminate enrolment. The policy (re)prompts the user about an item until either the DST returns a well-defined value or the top-level
policy reaches the limit for attempts (3; see Subsection 4.1). After exhausting all items, the policy terminates and writes the new profile into the KB.
4.3 Verification Models and Policies
Verification DST and Model. We track a verification score for each item  $s_{item}$  as follows (cf. Eq. 2):
$$
s _ {i t e m} = \operatorname {s c o r e} \left(\operatorname {i t e m} \left(p _ {\text {c l a i m e d}}\right), \operatorname {i t e m} \left(T _ {\text {d i a g o u g e}}\right)\right) \in [ 0, 1 ], \tag {4}
$$
The scores are initially undefined, and we track their maximum evaluation after each user input. For the experiments, we define the following scoring models: the random model samples from the [0,1] uniform distribution; the exact model returns 1 if the value from the claimed profile exactly matches any NLU n-best result, else, 0 (undefined for no NLU results); and the fuzzy model returns the best fuzzy match score between the value from the claimed profile and all NLU n-best results (undefined for no NLU results). We implement this as the normalised Levenshtein edit distance using the Wagner-Fischer algorithm (Wagner and Fischer, 1974). Finally, we evaluate a logical expression under fuzzy logic to combine all item-level scores (Eq. 4) into a profile-level score as follows (see Eq. 2):
$$
\begin{array}{l} s _ {p r o f i l e} = s _ {p o s t c o d e} \text {A N D} s _ {d o b} \text {A N D} \\ \left(s _ {\text {n a m e - f u l l}} \operatorname {O R} \left(s _ {\text {n a m e - f i r s t}} \text {A N D} s _ {\text {n a m e - l a s t}}\right)\right) \tag {5} \\ \end{array}
$$
Fuzzy logic (Zadeh, 1996) is a many-valued logic wherein truth values are real numbers in  $[0,1]$  that represent degrees of truthfulness and reasons using fuzzy logic operators (analogous to Boolean logic's AND, OR, and NOT). In the experiments, we choose the standard fuzzy logic operators (Zadeh, 1996):
$$
\begin{array}{l} \text {B o o l e a n} \longleftrightarrow \text {F u z z y} \\ \operatorname {A N D} (x, y) \longleftrightarrow \min  (x, y) \\ \mathrm {O R} (x, y) \longleftrightarrow \max  (x, y) \tag {6} \\ \operatorname {N O T} (x) \longleftrightarrow 1 - x \\ \end{array}
$$
Verification Policy. The task-level policy determines when to proceed to the next item, and decides when to terminate the verification process. The policy (re)prompts the user about an item until either the DST returns a well-defined score (Eq. 4) or the top-level policy reaches the limit for attempts (again, 3). The policy terminates either after exhausting all items or when it meets an early termination criterion: a low upper bound on the profile score (i.e. Eq. 5 with undefined  $\equiv$  1 is below the verification threshold,  $\theta$ ) guarantees a negative verification outcome. Upon termination, the policy returns the profile-level verification score (Eq. 5 with undefined  $\equiv$  0).
<sup>10</sup>https://cloud.google.com/text-to-speech
models	Profile	Postcode	Name	DoB
nlu	P%	R%	F1%	L	P%	R%	F1%	L	P%	R%	F1%	L	P%	R%	F1%	L
en-GB	cautious	38.83	30.27	34.02	4.15	69.08	55.20	61.37	1.83	65.88	64.88	65.38	1.12	80.37	78.97	79.66	1.21
seeking	27.44	23.34	25.22	3.86	59.90	51.16	55.18	1.70	63.74	63.51	63.63	1.10	63.86	63.58	63.72	1.07
pl-PL	cautious	66.41	60.37	63.25	3.98	95.51	91.91	93.68	1.51	71.86	69.26	70.54	1.20	92.92	90.31	91.59	1.26
seeking	53.07	51.63	52.34	3.69	87.85	86.44	87.14	1.38	69.76	69.16	69.46	1.20	82.83	82.37	82.60	1.11
fr-FR	cautious	34.22	30.37	32.19	3.85	77.62	72.09	74.75	1.50	44.21	44.00	44.10	1.06	90.81	86.81	88.76	1.29
seeking	26.46	24.68	25.54	3.63	75.03	70.43	72.66	1.46	44.27	44.19	44.23	1.06	72.12	71.57	71.84	1.10
4.4 Identification Models and Policies
Identification DST and Model. We track the NLU n-best results from all turns and the candidate profiles retrieved from the KB. Our identification process is an anytime algorithm (Zilberstein, 1996) that ranks the thus-far retrieved profiles by a score (Eq. 5), excluding profiles below an identification threshold,  $\theta$ . Following the literature on fuzzy retrieval (Zadrożny and Nowacka, 2009), instead of the standard fuzzy operators (Eq. 6), we use p-norm fuzzy operators (Salton et al., 1983):
$$
\mathrm {A N D} ^ {p} \left(s _ {1}, \dots , s _ {n}\right) = 1 - \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left| 1 - s _ {i} \right| ^ {p}\right) ^ {1 / p} \tag {7}
$$
$$
\mathrm {O R} ^ {p} (s _ {1}, \dots , s _ {n}) = \left(\frac {1}{n} \sum_ {i = 1} ^ {n} | s _ {i} | ^ {p}\right) ^ {1 / p}
$$
In the experiments, we approximate Eq. 7 by the infinity-one linear combination (Smith, 1990):
$$
\begin{array}{l} \mathrm {O R} _ {\alpha} = \alpha \mathrm {O R} ^ {\infty} + (1 - \alpha) \mathrm {O R} ^ {1} \\ = \alpha \max  + (1 - \alpha) \text {m e a n} \\ \mathrm {A N D} _ {\alpha} = \alpha \mathrm {A N D} ^ {\infty} + (1 - \alpha) \mathrm {A N D} ^ {1} \tag {8} \\ = \alpha \min  + (1 - \alpha) \text {m e a n} \\ \end{array}
$$
Note that  $\mathrm{AND}_1 = \mathrm{AND}^\infty = \min$  and  $\mathrm{OR}_1 = \mathrm{OR}^\infty = \max$  are the standard fuzzy operators (Eq. 6). Finally, an identification oracle always retrieves the correct profile if it is among the tracked candidates (i.e. retrieved from the KB).