Patent Description:
<CIT> relates to a system and method for biometric identification of a call originator. <CIT> relates to detecting user identity in shared audio source contexts.

Particular embodiments are set out in the independent claims. Various optional examples are set out in the dependent claims. It is an aspect of the present disclosure to provide a method according to claim <NUM>.

It is another aspect of the present disclosure to describe an intelligent voice assistant (IVA) system according to claim <NUM>.

Other aspects of the disclosure will be apparent from the following description and the appended claims.

Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

The disclosed system and method improve the accuracy of details recorded by intelligent voice assistant (IVA) systems, thereby reducing the time users spend on the phone. The disclosed system and method utilize phonetic (speech-sound based) representations at the transcription and alignment stages of the recognition process. The disclosed algorithm uses speech-to-phone (from caller audio) and text-to-phone (from existing written records) models to generate a shared auditory representation which can be used to identify caller (i.e., customer) information within the IVA system.

The individual speech sounds that distinguish spoken words are phonemes. A phoneme is therefore the smallest contrastive sound unit of a language. The individual letters or groups of letters that represent the individual speech sounds are called graphemes. Thus, a grapheme is the smallest fundamental unit in written language. In sum, phonemes represent sounds while graphemes include alphabetical letters, characters, numerical digits, and the like. In the embodiments described herein, written text in memory is stored in English and voice messages are spoken in English also. However, this is by way of example only and should not be construed to limit the scope of the disclosure and the claims herein. The teachings of the present disclosure may be used to process incoming calls from callers that speak languages other than English, or for information (e.g., names) based in a different phonemic system. In addition to phonemes, language acoustics can also be recorded as "phones", the smallest sound unit of a language (whether it distinguishes a word or not in that language). Phonemes are therefore a subset of phones. The disclosed system and method can be embodied using either sound representation. However, for the purposes of enabling the disclosed system and method and defining the scope of the claims herein as broadly as possible, the descriptions that follow are directed to an embodiment that uses phones to identify callers.

A primary implementation of the disclosed system and method is targeted at identifying the name of a caller. Specifically, the IVA system takes the audio of the caller pronouncing his or her name and transcribes International Phonetic Alphabet (IPA) phones. At the same time, the IVA system determines the list of potential names (i.e., candidate users) of the caller interacting with the IVA system based on the phone number of the calling phone or other device (e.g., caller ID) and generates an IPA transcription of the text-based candidate user names. Since the IVA system converts both the spoken name and written text names to a shared auditory space, the disclosed system and method then determines the most-likely customer name based on closest auditory similarity.

The disclosed system and method implement a contact prediction algorithm that includes four operations. In a first operation, the incoming caller audio is converted to IPA phones using a pre-trained phonetic transcription model (e.g., <NUM>-layer long short-term memory (LSTM) in a neural network architecture). In a second operation, a phone representation is derived from the text, via a grapheme-to-phone translation (e.g., a text-to-phone mapping, with ambiguities resolved based on part of speech tags, or a pre-trained ML model). In a third operation, spoken and textual originating phone representations are compared (in a one-to-many fashion). In a fourth operation, the closest customer name on record is identified based on a distance metric (e.g., Jaccard or Levenshtein distances).

The disclosed system and method are a novel and non-trivial combination of methods, which also leverage some of the advantages of data from the IVA system to make it viable. Specifically, although extracting the speech sounds of the caller audio and the customer text are two distinct and individually useful tasks, these operations have not been combined for record identification. This is because such systems would be less accurate relative to direct speech-to-text systems when applied to the task of transcription for generic language. However, record identification in particular is an ideal application for the disclosed system and method, because the out-of-vocabulary problem is highly common and because the potential transcriptions can be narrowed down to a smaller subset. In particular, the disclosed system and method use a caller's phone number to restrict the potential candidates, thereby making grapheme-to-phone transcription practical from a computational standpoint and reducing false positives from multiple candidate comparisons.

<FIG> shows a system <NUM> in accordance with disclosed embodiments. The system <NUM> comprises a call routing system <NUM> that receives incoming phone calls from caller devices via a communication network <NUM>. In an example embodiment, the call routing system <NUM> may part of an enterprise network <NUM> that is operated by a service provider, such as a software-as-a-service (SaaS) company. The enterprise network <NUM> also includes a customer records database (DB) <NUM> and one or more human agents stations, such as local human agent station 115A and remote human agent station 115B. Among other operations, the call routing system <NUM> may direct incoming calls to the local human agent station 115A within the enterprise network <NUM>. Alternatively, the call routing system <NUM> may direct incoming calls to the remote human agent station 115B via the communication network <NUM>.

The communication network <NUM> may be based on any conventional network architecture. In one example embodiment, the communication network <NUM> may be a circuit-switched system, such as a public switched telephone network (PSTN). In another example, the communication network <NUM> may be a packet-switched system, such as an Internet protocol (IP) network. In another example, the communication network <NUM> may comprise elements from both the PSTN and the Internet.

The calling devices may include conventional wireline phones, such as desk phones 145A, 145B, and 145C, which access the communication network <NUM> via a private branch exchange (PBX) <NUM>. The calling devices also may include mobile devices, such as mobile phones, tablets, and laptop computers. An example mobile device is mobile phone <NUM>, which accesses the communication network <NUM> via a wireless access point <NUM>. In an example embodiment, the wireless access point <NUM> may be a base station of a cellular telephone network.

The call routing system <NUM> comprises a switch <NUM> and an intelligent voice assistant (IVA) <NUM> that receive the incoming calls from communication network <NUM> and direct the incoming calls to the human agent stations 115A and 115B. The switch <NUM> may be any conventional switching fabric that comprises a combination of hardware and software that directs incoming calls received by call routing system <NUM> to the correct output port that communicates with the next node in the system <NUM>.

In an embodiment, the IVA <NUM> is configured to communicate with a caller using, for example, mobile phone <NUM> in order to prompt the caller to speak a predetermined word or phrase. In a typical embodiment, the IVA <NUM> prompts the caller to state his or her name. In alternate embodiments, the IVA <NUM> may ask the caller to state other unique information, such as a caller street address, a social security number, a phone extension in PBX <NUM>, and so forth. According to the principles of the present disclosure, the IVA <NUM> is configured to convert the spoken audio data of the caller into caller phones. The IVA <NUM> is further configured to use identifying information of the calling device (e.g., phone number) to access a customer record in customer records DB <NUM> and to retrieve or generate candidate user phones from the customer record for a plurality of candidate users. The IVA <NUM> then compares the caller phones to multiple ones of the candidate user phones to determine a candidate user having the closest matching candidate user phones. The IVA <NUM> then identifies the candidate user having the closest matching candidate user phones as the user.

<FIG> shows an intelligent voice assistant (IVA) <NUM> in accordance with disclosed embodiments. IVA <NUM> comprises processor <NUM> and server applications <NUM>. The server applications <NUM> include a plurality of controllers that comprise software applications executed by the processor <NUM> in order to perform the basic functions of IVA <NUM>. The server applications <NUM> include incoming call response controller <NUM>, caller voice capture controller <NUM>, customer records controller <NUM>, caller audio controller <NUM>, and audio comparison controller <NUM>.

The incoming call response controller <NUM> detects and answers an incoming call, identifies the calling device (e.g., mobile device <NUM>) using, for example, the phone number of the calling device or a network address associated with the calling device. The incoming call response controller <NUM> also identifies a customer account in the customer records DB <NUM> using the identifying information of the calling device. By way of example, the incoming call response controller <NUM> may use the calling device phone number to identify the correct customer account.

After the call is answered, the caller voice capture controller <NUM> prompts the caller to speak unique identifying information, such as the name of the caller, the caller's street address, the caller's social security number, and the like. The caller voice capture controller <NUM> is configured to record the spoken response of the caller for subsequent audio processing. The caller audio controller <NUM> is configured to convert the recorded spoken response of the caller into caller phones according to the principles of the present disclosure.

The customer records controller <NUM> accesses the customer account identified by the incoming call response controller <NUM> to retrieve from the identified customer account the text names of a plurality of users associated with the identified customer account. The customer records controller <NUM> is configured to convert the user text names to graphemes and then to convert the user graphemes to user phones.

Finally, the audio comparison controller <NUM> is configured to compare the caller phones from the incoming call to the user phones for the plurality of users associated with the identified customer account. The comparison operation determines the user phones that are the closest match to the caller phones using, for example, a distance metric (e.g., Jaccard, Levenshtein, etc.).

<FIG> shows a customer records database (DB) <NUM> in accordance with disclosed embodiments. The customer records database DB <NUM> includes records <NUM> for a plurality of customer accounts, such as customer account record 305A, customer account record 305B, and customer account record 305C. Each customer account record <NUM> comprises an account phone number <NUM> associated with the customer account record <NUM>. By way of example, the customer account record <NUM> for a business enterprise that owns PBX <NUM> stores the phone number of the PBX <NUM> in the account phone number <NUM>.

A large number of users may be associated with a customer account record 305A. By way of example, the business enterprise that owns PBX <NUM> may have <NUM> employees, each of which may be a user of the PBX <NUM>. In such a situation, the customer account record 305A may include a user record for each of the <NUM> users, such as user record 310A, user record 310B, and user record 310C. Each user record <NUM> includes a user information field <NUM>, such as example user information fields 312A, 312B, and 312C. The user information fields <NUM> contain information about each user, such as home or work address, job title, etc..

Each user record <NUM> also includes a name (text) field <NUM>, such as example name (text) fields 314A, 314B, and 314C. According to the principles of the disclosure, the IVA <NUM> retrieves the textual name data in the name (text) field <NUM> of a candidate user and converts the retrieved text to user graphemes that are subsequently converted to user phones. Optionally, each user record <NUM> also includes a user phone field <NUM>, such as example user phone fields 316A, 316B, and 316C. Once the textual name data of a candidate user is converted to candidate user phones, the candidate user phones may be stored in user phone field <NUM> for later use in subsequent phone calls. In an alternative embodiment of this system, phone field <NUM> could be populated from phones extracted from previous call audio.

<FIG> shows a flowchart in accordance with disclosed embodiments. In step <NUM>, the IVA <NUM> receives an incoming call and determines the calling device identification information associated with the call. In step <NUM>, the IVA <NUM> receives caller voice audio data from the caller. In step <NUM>, the IVA <NUM> converts the caller voice audio data to caller phones. In step <NUM>, the IVA <NUM> identifies a customer account associated with the calling device identification information.

In step <NUM>, the IVA <NUM> obtains user phones for multiple candidate users associated with the identified customer account. Next, in step <NUM>, the IVA <NUM> compares the caller phones to the user phones for the multiple candidate users. In step <NUM>, the IVA <NUM> determines the identity of the caller based on the results of the comparison.

As noted above, a primary implementation of the disclosed system and method identifies the name of a caller. Specifically, the IVA <NUM> captures the audio date of a caller pronouncing his or her name and transcribes International Phonetic Alphabet (IPA) phones (or phones). At the same time, the IVA <NUM> determines the candidate users of the customer account based on the phone number of the calling phone or other device (e.g., caller ID) and generates an IPA transcription of the text-based candidate user names. The IVA <NUM> converts both the spoken name of the caller and the written text names of the candidate users to a shared auditory space (i.e., phones). The disclosed system and method then determine the most-likely customer name based on closest auditory similarity.

<FIG> shows a flowchart in accordance with disclosed embodiments. In step <NUM>, the IVA <NUM> receives an incoming phone call and determines the phone number of the caller from the caller ID information. In step <NUM>, the IVA <NUM> prompts the caller to state the caller's full name. The IVA <NUM> records the spoken name audio data. In step <NUM>, the IVA <NUM> converts the recorded spoken name audio data into phones representing the spoken caller name.

In step <NUM>, the IVA <NUM> uses the caller phone number to identify a customer account associated with the caller. In step <NUM>, the IVA <NUM> retrieves multiple text names for multiple users associated with the identified customer account. In step <NUM>, the IVA <NUM> converts the multiple user text names into multiple name graphemes. In step <NUM>, the IVA <NUM> converts the multiple user name graphemes into multiple user name phones.

Optionally, the IVA <NUM> may store in the customer records database <NUM> user name phones that were generated during previous phone calls or during an account setup procedure. In such an optional embodiment, step <NUM> (indicated by dotted lines) may replace steps <NUM>, <NUM>, and <NUM>. In step <NUM>, the IVA <NUM> retrieves the stored user name phones from the database <NUM>.

In step <NUM>, the IVA <NUM> compares the caller name phones to multiple user name phones (either generated in step <NUM> or retrieved in step <NUM>) to identify a closest matching user name phone. The user associated with closest matching user name phone is identified as the caller. In step <NUM>, the IVA <NUM> forwards the call to a human agent station and forwards the user information of the identified caller to the human agent station 115A.

<FIG> show computing systems in accordance with disclosed embodiments. Embodiments disclosed herein may be implemented on a computing system. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be used. For example, as shown in <FIG>, the computing system (<NUM>) may include one or more computer processors (<NUM>), non-persistent storage (<NUM>) (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (<NUM>) (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (<NUM>) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities.

The computer processor(s) (<NUM>) may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or micro-cores of a processor. The computing system (<NUM>) may also include one or more input devices (<NUM>), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.

The communication interface (<NUM>) may include an integrated circuit for connecting the computing system (<NUM>) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and/or to another device, such as another computing device.

Further, the computing system (<NUM>) may include one or more output devices (<NUM>), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s). The input and output device(s) may be locally or remotely connected to the computer processor(s) (<NUM>), non-persistent storage (<NUM>), and persistent storage (<NUM>). Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms.

Software instructions in the form of computer readable program code to perform embodiments disclosed herein may be encoded, in whole or in part, temporarily or permanently, on a computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, carrier signals, transmission media, or any other computer readable medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments disclosed herein.

The computing system (<NUM>) in <FIG> may be connected to or be a part of a network. For example, as shown in <FIG>, the network (<NUM>) may include multiple nodes (e.g., node X (<NUM>), node Y (<NUM>)). Each node may correspond to a computing system, such as the computing system shown in <FIG>, or a group of nodes combined may correspond to the computing system shown in <FIG>. By way of an example, embodiments disclosed herein may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments disclosed herein may be implemented on a distributed computing system having multiple nodes, where each portion disclosed herein may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (<NUM>) may be located at a remote location and connected to the other elements over a network.

The nodes (e.g., node X (<NUM>), node Y (<NUM>)) in the network (<NUM>) may be configured to provide services for a client device (<NUM>). For example, the nodes may be part of a cloud computing system. The nodes may include functionality to receive requests from the client device (<NUM>) and transmit responses to the client device (<NUM>). The client device (<NUM>) may be a computing system, such as the computing system shown in <FIG>. Further, the client device (<NUM>) may include and/or perform all or a portion of one or more embodiments disclosed herein.

The computing system or group of computing systems described in <FIG> may include functionality to perform a variety of operations disclosed herein. For example, the computing system(s) may perform communication between processes on the same or different system. A variety of mechanisms, employing some form of active or passive communication, may facilitate the exchange of data between processes on the same device. Examples representative of these inter-process communications include, but are not limited to, the implementation of a file, a signal, a socket, a message queue, a pipeline, a semaphore, shared memory, message passing, and a memory-mapped file. Further details pertaining to a couple of these non-limiting examples are provided below.

Based on the client-server networking model, sockets may serve as interfaces or communication channel endpoints enabling bidirectional data transfer between processes on the same device. Foremost, following the client-server networking model, a server process (e.g., a process that provides data) may create a first socket object. Next, the server process binds the first socket object, thereby associating the first socket object with a unique name and/or address. After creating and binding the first socket object, the server process then waits and listens for incoming connection requests from one or more client processes (e.g., processes that seek data). At this point, when a client process wishes to obtain data from a server process, the client process starts by creating a second socket object. The client process then proceeds to generate a connection request that includes at least the second socket object and the unique name and/or address associated with the first socket object. The client process then transmits the connection request to the server process. Depending on availability, the server process may accept the connection request, establishing a communication channel with the client process, or the server process, busy in handling other operations, may queue the connection request in a buffer until server process is ready. An established connection informs the client process that communications may commence. In response, the client process may generate a data request specifying the data that the client process wishes to obtain. The data request is subsequently transmitted to the server process. Upon receiving the data request, the server process analyzes the request and gathers the requested data. Finally, the server process then generates a reply including at least the requested data and transmits the reply to the client process. The data may be transferred, more commonly, as datagrams or a stream of characters (e.g., bytes).

Shared memory refers to the allocation of virtual memory space in order to substantiate a mechanism for which data may be communicated and/or accessed by multiple processes. In implementing shared memory, an initializing process first creates a shareable segment in persistent or non-persistent storage. Post creation, the initializing process then mounts the shareable segment, subsequently mapping the shareable segment into the address space associated with the initializing process. Following the mounting, the initializing process proceeds to identify and grant access permission to one or more authorized processes that may also write and read data to and from the shareable segment. Changes made to the data in the shareable segment by one process may immediately affect other processes, which are also linked to the shareable segment. Further, when one of the authorized processes accesses the shareable segment, the shareable segment maps to the address space of that authorized process. Often, only one authorized process may mount the shareable segment, other than the initializing process, at any given time.

Other techniques may be used to share data, such as the various data described in the present application, between processes without departing from the scope of the disclosure. The processes may be part of the same or different application and may execute on the same or different computing system.

The computing system in <FIG> may implement and/or be connected to a data repository. For example, one type of data repository is a database. A database is a collection of information configured for ease of data retrieval, modification, re-organization, and deletion. Database Management System (DBMS) is a software application that provides an interface for users to define, create, query, update, or administer databases.

The user, or software application, may submit a statement or query into the DBMS. Then the DBMS interprets the statement. The statement may be a select statement to request information, update statement, create statement, delete statement, etc. Moreover, the statement may include parameters that specify data or data container (database, table, record, column, view, etc.), identifier(s), conditions (comparison operators), functions (e.g., join, full join, count, average, etc.), sort (e.g., ascending, descending), or others. The DBMS may execute the statement. For example, the DBMS may access a memory buffer, a reference or index a file for read, write, deletion, or any combination thereof, for responding to the statement. The DBMS may load the data from persistent or non-persistent storage and perform computations to respond to the query. The DBMS may return the result(s) to the user or software application.

The above description of functions presents only a few examples of functions performed by the computing system of <FIG> and the nodes and/or client device in <FIG>. Other functions may be performed using one or more embodiments disclosed herein.

Thus, from one perspective, there has now been described a method including receiving an incoming call from a calling device of a caller and determining identification information for the calling device. The method also includes receiving voice audio data of the caller from the calling device, converting the voice audio data to caller phones, and identifying a customer account associated with the identification information. The method further includes obtaining user phones for multiple candidate users associated with the identified customer account, comparing the caller phones to the user phones for the multiple candidate users, and determining the identity of the caller based on the comparison.

Claim 1:
A method comprising:
receiving (<NUM>) an incoming call from a calling device of a caller:
determining (<NUM>) identification information for the calling device;
receiving (<NUM>) voice audio data of the caller from the calling device;
converting (<NUM>) the voice audio data to caller phones:
identifying (<NUM>) a customer account associated with the identification information;
obtaining (<NUM>) user phones for a plurality of candidate users associated with the identified customer account, wherein obtaining user phones for the plurality of candidate users further comprises converting user text data associated with each of the plurality of candidate users to user graphemes and converting a user grapheme to a user phone for each of the plurality of candidate users;
comparing (<NUM>) the caller phones to the user phones for the plurality of candidate users; and
determining (<NUM>) the identity of the caller based on the comparison, wherein determining the identity of the caller based on the comparison comprises identifying a closest user name based on a distance metric.