Audio verification

Techniques for enabling a system to verify that contact data are to be added to a user account are described. A system receives message data. The system generates first audio data that includes a representation of a verification code. The system sends a message comprising the audio data to a first device. The system receives, from a second device, second audio data that represents the first audio data. The system confirms that the second audio data included a representation of the verification code, and then verifies that the contact data is to be added to the user account.

BACKGROUND

A large and growing population of users is enjoying entertainment through the consumption of digital media items, such as music, movies, images, electronic books, and so on. The users employ various electronic devices to consume such media items. Among these electronic devices (referred to herein as user devices or user equipment) are electronic book readers, cellular telephones, personal digital assistants (PDAs), portable media players, tablet computers, netbooks, laptops, and the like. These electronic devices wirelessly communicate with a communications infrastructure to enable the consumption of the digital media items and to perform other operations such as telephony operations. In order to access content and/or use some functions of such an electronic device, the electronic device is generally registered to a user account, and contact data is associated with the user account.

DETAILED DESCRIPTION

Embodiments described herein are directed to verification of operations using audio data. In embodiments, verification may be performed to confirm that an entity that initiated an operation has authorization to complete the operation. Examples include verification that a phone number is controlled by a user associated with a user account, verification of a purchase transaction, verification that a requestor has access to information, verification that a requestor has access to enter a location, verification that contact data is to be added to a user account, and so on. For example, embodiments may verify that a correct phone number is linked to a user account (e.g., that the user requesting to link the phone number to their user account actually owns or controls the phone number). Embodiments take advantage of the capabilities of a speech-detection device to facilitate verification of an operation with minimal user input, improving a user experience for operations such as verification of phone numbers for user accounts.

Speech recognition systems have progressed to the point where humans can interact with computing devices using their voices. Such systems employ techniques to identify the words spoken by a human user based on the various qualities of a received audio input. Speech recognition combined with natural language understanding processing techniques enable speech-based user control of a computing device to perform tasks based on the user's spoken commands. The combination of speech recognition and natural language understanding processing techniques is referred to herein as speech processing. Speech processing may also involve converting a user's speech into text data which may then be provided to various text-based software applications. Speech processing may be used by computers, hand-held devices, telephone computer systems, kiosks, and a wide variety of other devices to improve human-computer interactions.

Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics concerned with transforming audio data associated with speech into text data representative of that speech. Natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics concerned with enabling computers to derive meaning from text input containing natural language. ASR, NLU may be used together as part of a speech processing system.

A user may request verification of a phone number (e.g., request that a phone number be added to a user account) via an application executing on a first device. The application may be, for example, a speech-detection application. Each instance of the application may have a unique application identifier (ID). The first device may be a mobile phone associated with a phone number. Responsive to the request, the application may generate a text message (e.g., an SMS message) and send the text message to a server. The server may then generate a verification code, encode the verification code into audio data, and send a response message to the phone number associated with the first device.

The user may speak an utterance (e.g., a command) to a second device that is part of a speech processing system configured to execute one or more commands corresponding to input speech. For example, the second device may capture audio corresponding to “device, verify phone number,” “device, verify transaction,” or the like. The utterance may include an activation code that places the second device into a listening mode, in which it captures audio and sends the captured audio to the speech processing system. The speech processing system may process the utterance to determine that a verification operation has been requested.

The user may cause the first device to play the audio data, which may be captured by the second device. The speech processing system may process the captured audio to extract the audio verification code from the audio data in the captured audio. The captured audio may be an audio recording or an audio stream, for example. A result of the extraction is a text-based version of the verification code. The speech processing system may then verify that the phone number should be added to the user account of the user. Similar techniques may also be used to verify other operations, information, transactions, and so on.

The present disclosure provides new techniques for verifying phone numbers, transactions, other operations, and information using audio data. These techniques reduce the amount of information that a user is expected to manually input, and may reduce user input to a press of one or a few buttons while maintaining maximum security.

FIG. 1Ashows a system100configured to verify operations and/or contact data using audio data. Although the figures and discussion illustrate certain operations of the system100in a particular order, the operations described may be performed in a different order (as well as certain operations removed or added) without departing from the intent of the disclosure. As shown inFIG. 1A, a first device110local to a user5, servers120a-b, a communication router device125, and a second device115local to the user5may be in communication over one or more networks199. The servers120a-b(which may be one or more different physical devices) may be capable of performing traditional speech processing (e.g., ASR, NLU, command processing, etc.) as well as other operations as described herein. A single server120amay perform all speech processing or multiple servers120a-bmay combine to perform all speech processing. Further, the servers120a-bmay execute certain commands, such as answering spoken utterances of users5, sending messages of users5, and operating other devices (e.g., light switches, appliances, etc.). In one embodiment, servers120a-binclude a first server120aor set of servers that perform speech processing and a second server120bor set of servers that include a communications system. Alternatively, a single server may include both a speech processing system and a communications system. In addition, certain speech-detection or command execution functions may be performed by the first device110, which may be a speech-detection device.

As shown inFIG. 1A, server(s)120b(or server120a) may receive message data from a particular phone number (block142). The message data may be received in the form of a text message (e.g., an SMS message or a multimedia messaging system (MMS) message) generated by device115and sent to a phone number associated with server120b. An application running on the device115may generate the message data and use an application programming interface (API) of the device115to send the message data using a messaging protocol enabled on the device115. The application, which may be for example a speech-detection application, may generate message content and then send a request or instruction to a messaging application on the device115(e.g., using an API for the messaging application) to generate and send a message to a particular destination (e.g., to a phone number used by server120b) to trigger phone number verification operations.

The message data may be sent to communication router devices125. The communication router devices125may include one or more systems of a cellular service provider. Such systems may include, for example, a messaging system controlled by a cellular phone service provider, such as a short message service (SMS) system or a multimedia messaging service (MMS) system. Other examples of protocols that may be used to send and receive the message data include hypertext transport protocol (HTTP), hypertext transport protocol secure (HTTPS), simple object access protocol (SOAP), and so on. The message data may include a header and a payload, which may vary based on the messaging protocol used. The message data may be synchronous message data or asynchronous message data. For example, the message data may be in the form of a phone call or video call to a phone number associated with server120b. The messaging system may send the message data on to server120b.FIG. 1Billustrates transmission of message data from the device115to the server(s)120a-b(block143). The message data may be message data that was manually typed by a user (e.g., using an SMS or MMS application on device115). Alternatively, an application on the device115(e.g., a speech-detection application) may generate text and auto populate a message, along with a recipient address or phone number. The user may then be asked to press the send button to send the message data. Alternatively, the device115may automatically send the message data.

Receipt of the message data by the server120bmay trigger a verification service and/or verification domain on the server120b. Responsive to receiving the message data, the server120bmay generate a verification code (block144). The verification code may be a random number generated using, for example, a random number generator or pseudo-random number generator. As used herein, the term random number includes both completely random numbers and pseudorandom numbers. In one embodiment, a hash is performed on the random number to generate the verification code. The verification code may have any number of digits, some examples being 6 digits, 10 digits, 15 digits, 32 digits, 64 digits, and so on. A greater number of digits increases the security of the verification code. A non-audio or text (e.g., binary, numeric or alphanumeric) representation of the verification code may then be stored (also referred to as a non-audio instance of the verification code and non-audio data representing the verification code).

The server120bmay then generate audio data representing the verification code (block146). This may include generating a new audio file and/or encoding audio information representing the verification code into an existing audio file. The verification code may be encoded into an audio format using one or more audio modulation schemes such as frequency shift keying, phase shift keying, pulse modulation, and so on. The verification code may also be encoded into an audio format using text to speech. For example, an audio code of 123 could be encoded into an audio file of a voice sayings “1 2 3”. One or both of the server(s)120a-bmay include an encoder that encodes the verification code into the audio data (e.g., that generates the audio data representing the verification code) and/or a decoder that will later decode the verification code from the audio data (e.g., that generates a text version of the verification code from the audio data).

Once the audio data is generated, the server120bmay generate message data (e.g., a response message) comprising the audio data (e.g., audio verification data) or a link to the audio data, and sends the message data to the phone number from which the initial message data was received. The communication router devices125may then send the message data to the device associated with the phone number, which is the device115that generated the initial text message. The message data may be included in a text message (e.g., an SMS or MMS message) that includes the audio data or a link to the audio data. Other examples of protocols that may be used to send and receive the message data include hypertext transport protocol (HTTP), hypertext transport protocol secure (HTTPS), simple object access protocol (SOAP), and so on. The message data may include a header and a payload, which may vary based on the messaging protocol used.FIG. 1Cillustrates transmission of the message data from the server(s)120a-bto the device115(block149).

A device110may be a speech-detection device that captures audio11including a spoken utterance of a user5via a microphone or microphone array of the device110. The audio11may include a voice activation command for the device110, which may place the device110into a listening mode. While in the listening mode, the device110activates one or more microphones and sends captured audio to one or more servers120a-b. The device110determines audio data corresponding to the captured audio11, and sends the audio data to the server(s)120a-bfor processing.

While the device110is in the listening mode, the device115plays the audio data12that was included in the response message (e.g., included as a file or a hyperlink in the response message). The device110captures the audio data12and sends second audio data including the captured audio data12to the server(s)120a-b. The server(s)120a-breceives (150) the second audio data from the device110. The server(s)120a-bmay receive the captured audio in the form of an audio recording or an audio stream, which may include the audio data12as well as background noise, additional utterances of the user5, and/or other sounds.FIG. 1Dillustrates device115playing the audio data12to device110.FIG. 1Eillustrates device110sending the second audio data (captured audio including the audio data12and possibly other sounds) to the server(s)120a-b(block151).

In addition to the audio11containing a command that places the device110into listening mode, the audio11may additionally include a further command to perform a verification operation. For example, the audio11may contain the phrase, “Device X, verify.” The utterance “device X” may place the device110into listening mode, while the phrase “verify” may be a command that will initiate a verification operation at one or more of the servers120a-b.

The server120amay determine, based on speech processing (e.g., ASR and NLU) of the audio data, that the utterance corresponds to a command to verify a phone number (or verify another operation or other information). The server120aor server120bextracts the verification code from the second audio data (captured audio) (block152). This may include storing the captured audio, and processing the captured audio using a decoder. The decoder may convert the audio data into non-audio data (e.g., text/numerical data) comprising the verification code. The server120bmay then verify that the device115is associated with the phone number to which the server120bsent the message data (response message) (block154). Verification may be performed by comparing a stored version of the verification code to the version of the verification code that was decoded from the second audio data. If the two verification codes have the same value, then the phone number may be confirmed. The server120bmay therefore verify the phone number for a user account associated with the device, which may include adding the phone number to the user account. At this point, an application on the device115is able to use the phone number for one or more actions, such as to make or receive phone calls using a voice over internet protocol (VOIP) service.

The system100ofFIGS. 1A-1Emay operate using various speech processing components as described inFIGS. 2A and 2Bas well as a communications system as described inFIG. 2C. The various components illustrated may be located on a same or different physical devices. Communication between various components illustrated inFIGS. 2A-Cmay occur directly or across a network(s)199. An audio capture component, such as a microphone (or array of microphones) of the speech-detection device110(or other device), captures input audio11corresponding to a spoken utterance. The speech-detection device110, using a wakeword detection component220, processes audio data corresponding to the input audio11to determine if a keyword (such as a wakeword) is detected in the audio data. Following detection of a wakeword, the speech-detection device110sends audio data111, corresponding to the utterance, to a server(s)120a-bfor processing. The audio data111may be output from an acoustic front end (AFE)256located on the speech-detection device110prior to transmission, or the audio data111may be in a different form for processing by a remote AFE256, such as the AFE256located with an ASR component250of the server(s)120.

The wakeword detection component220works in conjunction with other components of the speech-detection device110, for example a microphone to detect keywords in audio data corresponding to the input audio11. For example, the speech-detection device110may convert input audio11into audio data, and process the audio data with the wakeword detection component220to determine whether speech is detected, and if so, if the audio data comprising speech matches an audio signature and/or model corresponding to a particular keyword, (e.g., “verify”).

The speech-detection device110may use various techniques to determine whether audio data includes speech. Some embodiments may apply voice activity detection (VAD) techniques. Such techniques may determine whether speech is present in audio data based on various quantitative aspects of the audio data, such as a spectral slope between one or more frames of the audio data; energy levels of the audio data in one or more spectral bands; signal-to-noise ratios of the audio data in one or more spectral bands; or other quantitative aspects. In other embodiments, the speech-detection device110may implement a limited classifier configured to distinguish speech from background noise. The classifier may be implemented by techniques such as linear classifiers, support vector machines, and decision trees. In still other embodiments, Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques may be applied to compare the audio data to one or more acoustic models in speech storage, which acoustic models may include models corresponding to speech, noise (such as environmental noise or background noise), or silence. Still other techniques may be used to determine whether speech is present in audio data.

Once speech is detected in the audio data, the speech-detection device110may use the wakeword detection component220to perform wakeword detection to determine when a user intends to speak a command to the speech-detection device110. This process may also be referred to as keyword detection, with the wakeword being a specific example of a keyword. Specifically, keyword detection is typically performed without performing linguistic analysis, textual analysis, or semantic analysis. Instead, audio data is analyzed to determine if specific characteristics of the audio data match preconfigured acoustic waveforms, audio signatures, or other data to determine if the incoming audio data “matches” stored audio data corresponding to a keyword.

Once the wakeword is detected in the audio data, the speech-detection device110may “wake” and begin transmitting audio data111corresponding to input audio11to server120afor speech processing (e.g., for purposes of executing a command in the speech). The audio data111may include data corresponding to the wakeword, or the portion of the audio data corresponding to the wakeword may be removed by the speech-detection device110prior to sending the audio data111to the server120a. The audio data may additionally include a voice command associated with verification of an operation, verification of contact data and/or an audio verification code.

Upon receipt by the server120a, an orchestrator component297sends the audio data111to a speech processing component298, and namely an ASR component250of the speech processing component298. The ASR component250transcribes the audio data111into text data representing words of speech contained in the audio data111. The text data may then be used by other components for various purposes, such as executing system commands, inputting data, etc. A spoken utterance in the audio data111is input to a processor configured to perform ASR, which then interprets the spoken utterance based on a similarity between the spoken utterance and pre-established language models254stored in an ASR model knowledge base (i.e., an ASR model storage252). For example, the ASR component250may compare the audio data111with models for sounds (e.g., subword units or phonemes) and sequences of sounds to identify words that match the sequence of sounds spoken in the utterance of the audio data111.

The different ways a spoken utterance may be interpreted (i.e., the different hypotheses) may each be assigned a respective probability/confidence score representing a likelihood that a particular set of words matches those spoken in the utterance. The confidence score may be based on a number of factors including, for example, a similarity of the sound in the spoken utterance to models for language sounds (e.g., an acoustic model253stored in the ASR model storage252), and a likelihood that a particular word that matches the sound would be included in the sentence at the specific location (e.g., using a language model254stored in the ASR model storage252). Thus, each potential textual interpretation of the spoken utterance (i.e., hypothesis) is associated with a confidence score. Based on the considered factors and the assigned confidence score, the ASR component250outputs the most likely text data corresponding to the audio data111. The ASR component250may also output multiple hypotheses in the form of a lattice or an N-best list with each hypothesis corresponding to a confidence score or other score (e.g., such as probability scores, etc.).

The server120aincluding the ASR component250may include an AFE256and a speech recognition engine258. The AFE256transforms audio data111into data for processing by the speech recognition engine258. The speech recognition engine258compares the speech recognition data with acoustic models253, language models254, and other data models and information for recognizing the speech conveyed in the audio data111. The AFE256may reduce noise in the audio data111and divide the digitized audio data111into frames representing time intervals for which the AFE256determines a number of values (i.e., features) representing qualities of the audio data111, along with a set of those values (i.e., a feature vector or audio feature vector) representing features/qualities of the audio data111within each frame. In one configuration, each audio frame includes 25 ms of audio data and the frames start at 10 ms intervals resulting in a sliding window where adjacent audio frames include 15 ms of overlapping audio data. Many different features may be determined, as known in the art, and each feature represents some quality of the audio data111that may be useful for ASR processing. A number of approaches may be used by the AFE256to process the audio data111, such as mel-frequency cepstral coefficients (MFCCs), perceptual linear predictive (PLP) techniques, neural network feature vector techniques, linear discriminant analysis, semi-tied covariance matrices, or other approaches known to those skilled in the art.

The speech recognition engine258may process data output from the AFE256with reference to information stored in the ASR model storage252. Alternatively, post-AFE processed data (e.g., feature vectors) may be received by the device executing ASR processing from another source besides the internal AFE256. For example, the speech-detection device110may process audio data111into feature vectors (e.g., using an on-device AFE256) and transmit the feature vector data to the server120a, across the network(s)199, for ASR processing. Feature vector data may arrive at the server120aencoded, in which case it may be decoded prior to processing by the processor executing the speech recognition engine258.

The speech recognition engine258attempts to match received feature vectors to language phonemes and words as known in the stored acoustic models253and language models254. The speech recognition engine258computes recognition scores for the feature vectors based on acoustic information and language information. The acoustic information is used to calculate an acoustic score representing a likelihood that the intended sound represented by a group of feature vectors matches a language phoneme. The language information is used to adjust the acoustic score by considering what sounds and/or words are used in context with each other, thereby improving a likelihood that the ASR component250will output text data representing speech that makes sense grammatically.

The speech recognition engine258may use a number of techniques to match feature vectors to phonemes, for example using HMMs to determine probabilities that feature vectors may match phonemes. Sounds received may be represented as paths between states of the HMM and multiple paths may represent multiple possible text matches for the same sound.

Following ASR processing, the ASR results (i.e., text data representing speech) may be sent by the speech recognition engine258to the orchestrator297or other processing components, which may be local to the device performing ASR and/or distributed across the network(s)199. For example, ASR results in the form of a single textual representation of the speech, an N-best list including multiple hypotheses and respective scores, lattice, etc. may be sent from the ASR component250to an NLU component260either directly or indirectly through the orchestrator component297.

The device performing NLU processing (e.g., the server120a) may include various components, including potentially dedicated processor(s), memory, storage, etc. The device performing NLU processing may include a dedicated NLU component260, which may include a named entity recognition (NER) component262and an intent classification (IC) component264. The device performing NLU processing may additionally include NLU storage273, and a knowledge base (not illustrated). The knowledge base is a database or other information storage that may include information about entities that may be used in resolving spoken utterances. The NLU component260may also utilize gazetteer information284stored in an entity library storage282. The knowledge base and/or gazetteer information284may be used for entity resolution, for example matching ASR results with different entities (e.g., song titles, contact names, etc.). Gazetteers284may be linked to users (e.g., a particular gazetteer may be associated with a specific user's music collection), may be linked to certain domains (e.g., shopping), or may be organized in a variety of other ways. Domain, as used herein, may refer to a category of content, such as music, videos, weather, etc.

The NLU component260takes text data (e.g., output from the ASR component250) and attempts to make a semantic interpretation of the text data. That is, the NLU component260determines the meaning behind the text data based on the individual words and then implements that meaning. The NLU component260interprets a text string to derive an intent or a desired action from the user as well as the pertinent pieces of information in the text data that allow a device (e.g., the speech-detection device110, the server120a, server communication router125, etc.) to complete that action. For example, if a spoken utterance is processed using the ASR component250, which outputs the text data “call mom”, the NLU component260may determine the user intended to activate a telephone in his/her device and to initiate a call with a contact matching the entity “mom.”

The NLU component260may process several textual inputs related to the same utterance. For example, if the ASR component250outputs N text segments (e.g., as part of an N-best list), the NLU component260may process all N outputs to obtain NLU results.

The NLU component260may be configured to parse and tag to annotate text data as part of NLU processing. For example, for the text data “verify phone number,” the NLU component260may tag “verify” as a command (e.g., to execute verification of a phone number) and may tag “phone number” as a specific entity and target of the command.

To correctly perform NLU processing of an utterance, the NLU component260may be configured to determine a “domain” of the utterance so as to determine and narrow down which services offered by an endpoint device (e.g., the server(s)120, the speech-detection device110, a communication router125, etc.) may be relevant. For example, an endpoint device may offer services relating to interactions with a telephone service, a contact list service, a calendar/scheduling service, a music player service, a verification service, etc. Words in text data may implicate more than one service, and some services may be functionally linked (e.g., both a telephone service and a calendar service may utilize data from the contact list).

The NER component262receives an utterance in the form of ASR results and attempts to identify relevant grammars and lexical information that may be used to construe meaning. To do so, the NER component262may begin by identifying potential domains that may relate to the received utterance. The NLU storage273includes a database of domains274associated with specific devices. For example, the speech-detection device110may be associated with domains for music, telephony, calendaring, contact lists, verifications, and device-specific communications. In addition, the entity library282may include database entries about specific services on a specific device, either indexed by Device ID, User ID, Household ID, or some other indicator.

A domain may represent a discrete set of activities having a common theme, such as “shopping”, “music”, “calendaring”, “verification,” etc. As such, each domain may be associated with a particular language model and/or grammar database276, a particular set of intents/actions278, and/or a particular personalized lexicon286. Each gazetteer284may include domain-indexed lexical information associated with a particular user and/or device. For example, the Gazetteer A284aincludes domain-index lexical information286aato286an. A user's music-domain lexical information might include album titles, artist names, and song names, for example, whereas a user's contact-list lexical information might include the names of contacts. Since every user's music collection and contact list is presumably different, this personalized information improves entity resolution.

An utterance may be processed applying the rules, models, and information applicable to each identified domain. For example, if an utterance potentially implicates both communications and music, the utterance may be NLU processed using the grammar models and lexical information for communications, and may also be processed using the grammar models and lexical information for music. The responses to the spoken utterance produced by each set of models is scored, with the overall highest ranked result from all applied domains being ordinarily selected to be the correct result.

The IC component264parses the utterance to determine an intent(s) for each identified domain, where the intent(s) corresponds to the action to be performed that is responsive to the spoken utterance. Each domain is associated with a database278of words linked to intents. For example, a music intent database may link words and phrases such as “quiet,” “volume off,” and “mute” to a “mute” intent. A verification database may link words and phrases such as “verify operation,” “verify transaction”, “verify phone number”, “pair device”, “pair phone number”, “verify”, and so on. The IC component264identifies potential intents for each identified domain by comparing words in the utterance to the words and phrases in the intents database278.

In order to generate a particular interpreted response, the NER component262applies the grammar models and lexical information associated with the respective domain. Each grammar model276includes the names of entities (i.e., nouns) commonly found in speech about the particular domain (i.e., generic terms), whereas the lexical information286from the gazetteer284is personalized to the user and/or the device. For instance, a grammar model276associated with a verification domain may include a database of words commonly used when people request verification or confirmation of data, transactions, or operations.

The intents identified by the IC component264are linked to domain-specific grammar frameworks (included in276) with “slots” or “fields” to be filled. For example, if “verify phone number” is an identified intent, a grammar framework(s) may correspond to sentence structures such as “Verify {operation},” “Verify {data},” “Verify {transaction},” “Verify {phone number},” etc. However, to make recognition more flexible, these frameworks would ordinarily not be structured as sentences, but rather based on associating slots with grammatical tags.

For example, the NER component262may parse the spoken utterance to identify words as subject, object, verb, preposition, etc., based on grammar rules and models, prior to recognizing named entities. The identified verb may be used by the IC component264to identify intent, which is then used by the NER component262to identify frameworks. A framework for an intent of “verify” may specify a list of slots/fields applicable to verify the identified “object” and any object modifier (e.g., a prepositional phrase), such as {my}, {user name}, {first}, etc. The NER component262then searches the corresponding fields in the domain-specific and personalized lexicon(s), attempting to match words and phrases in the utterance tagged as a grammatical object or object modifier with those identified in the database(s).

This process includes semantic tagging, which is the labeling of a word or combination of words according to their type/semantic meaning. Parsing may be performed using heuristic grammar rules, or the NER component262may be constructed using techniques such as HMMs, maximum entropy models, log linear models, conditional random fields (CRF), and the like.

For instance, an utterance of “verify phone number” might be parsed and tagged as {Verb}: “Verify,” {Object}: “phone number.” At this point in the process, “Verify” is identified as a verb based on a word database associated with the verification domain, which the IC component264will determine corresponds to the “verify phone number” intent.

The frameworks linked to the intent are then used to determine what database fields should be searched to determine the meaning of these phrases, such as searching a user's gazetteer for similarity with the framework slots. So a framework for a “Verify” intent might indicate to attempt to resolve the identified object based on {object name}, {transaction label}, and/or {operation identifier}. If the search of the gazetteer does not resolve the slot/field using gazetteer information, the NER component262may search the database of generic words associated with the domain (i.e., in the NLU storage273). For instance, if the utterance was “play songs by the rolling stones,” after failing to determine an album name or song name called “songs” by “the rolling stones,” the NER component262may search the domain vocabulary for the word “songs.” In the alternative, generic words may be checked before the gazetteer information, or both may be tried, potentially producing two different results.

The comparison process used by the NER component262may classify (i.e., score) how closely a database entry compares to a tagged word or phrase, how closely the grammatical structure of the utterance corresponds to the applied grammatical framework, and based on whether the database indicates a relationship between an entry and information identified to fill other slots of the framework.

The NER component262may also use contextual operational rules to fill slots. For example, if a user previously sent message data that triggered a verification code to be generated, the NER component262may apply an inference-based rule to fill a slot associated with the verification code that was generated.

The results of NLU processing may be tagged to attribute meaning to the utterance. For example, “Verify phone number” might produce a result of: {domain} Verification, {intent} Verify, {object to verify} “phone number.”

The output from the ASR component250may also be sent to a user recognition component295either directly or indirectly through the orchestrator component297. Alternatively, the user recognition component295may be implemented as part of the ASR component250. The user recognition component295performs user recognition using the audio data111, and optionally the ASR component output. The user recognition component295may include a scoring component that determines respective scores indicating whether the input utterance in the audio data111was spoken by particular users. The user recognition component2may also include a confidence component that determines an overall confidence as the accuracy of user recognition operations. User recognition may involve comparing speech characteristics in the audio data111to stored speech characteristics of users. User recognition may also involve comparing biometric data (e.g., fingerprint data, iris data, etc.) received by the user recognition component295to stored biometric data of users. User recognition may further involve comparing image data including a representation of a feature of a user with stored image data including representations of features of users. It should be appreciated that other kinds of user recognition processes, including those known in the art, may be used.

Output from the NLU processing, which may include tagged text data, commands, etc., and output of the user recognition component295(e.g., a unique ID of a user) may be sent to a command processor290, which may be located on a same or separate server120a-bas part of the system100. The system100may include more than one command processor290, and the command processor(s)290may be determined based on the NLU output. For example, if the NLU output includes a command to play music, the command processor290selected may correspond to a music playing application, such as one located on the speech-detection device110or in a music playing appliance. In another example, if the NLU output includes a command to verify an operation, the command processor290selected may correspond to a verification domain or verification service. Many such command processors290may be available to the system100depending on the various applications that may be invoked. In one embodiment, a command processor290amay be associated with a verification service and/or communication system that may be located on a different server (e.g., server120b). The command processor290amay be an interface for sending data to the verification service and/or communication system located on the different server. The communication system is described in greater detail below with reference toFIG. 2C.

Output from the NLU component260(and optionally the user recognition component295) may be sent to a command processor(s)290/skill either directly or indirectly via the orchestrator component297. A “skill” may correspond to a domain and may be software running on a server or device akin to an application. That is, a skill may enable a server to execute specific functionality in order to provide data or produce some other output called for by a user. The system may be configured with more than one skill. For example a weather service skill may enable a server to execute a command with respect to a weather service server, a car service skill may enable a server to execute a command with respect to a taxi service server, an order pizza skill may enable a server to execute a command with respect to a restaurant server, a verification skill may enable a server to execute a command with respect to a verification service, etc.

The server120amay include multiple command processor(s)290a,290bthrough290n. These command processors290a-nmay include, for example, a communication manager command processor290aand a verification command processor290b. The communication manager command processor290amay be configured to perform various operations described herein with respect to message generation and transmission, or may be configured to interface with a communications system. The verification command processor290bmay be configured to perform various operations described herein with respect to verification of an operation such as a purchase transaction or linking of a phone number to a user account, or may be configured to interface with a verification system. A verification skill or system may provide a verification service for maintaining open verification requests, comparing verification codes, decoding audio data to extract verification codes from the audio data, and notifying other skills, domains or command processors290a-nwhen a verification is completed successfully.

A command processor290a-nmay output text that is to be spoken to a user via speech-detection device110. Alternatively, a command processor290a-nmay receive text from, for example, a communication service, where the text is to be spoken to a user via speech-detection device110. For example, a verification command processor may receive or output text stating, “please play the audio verification code.” The command processor290a-nprovides the text to a text to speech (TTS) engine214, which converts the text into speech (e.g., into an audio file that contains a spoken version of the content of the text). This audio file may be, for example, a Moving Picture Experts Group Audio Layer III (MP3) audio file or other type of compressed or uncompressed audio file. The audio file is sent to the speech-detection device110and then played by the speech-detection device110. In one embodiment, a link to the audio file (e.g., a universal resource locator (URL)) is sent to the speech-detection device, and the speech-detection device110accesses the link to receive the audio file.

In addition to outputting text, for example, to “play audio verification code,” the verification command processor may place the orchestrator into a state in which subsequent audio data received by server(s)120a-bis sent directly to a particular command processor (e.g., to the verification command processor) or other component (e.g., to the communications system). In one embodiment, the verification command processor may transfer control of the speech-detection device110to the communications system. Audio data may then be sent directly to the communications system, bypassing server120auntil control is returned to server120a.

After the speech-detection device110outputs audio stating, for example, “play audio verification code”, a user device may play an audio verification code that was sent to the user device within a threshold proximity of the speech-detection device110(e.g., within a distance in which the speech-detection device110can pick up or detect the playing of the audio verification code). Audio data representing the played audio verification code may be received at server120aand forwarded by orchestrator297directly to the appropriate command processor290a-nor other component, bypassing the speech processing unit298. The command processor290a-nmay include an audio decoder that processes the audio and decodes the audio into a non-audio (e.g., text or alphanumeric) representation of the verification code. The command processor290a-nmay then forward the non-audio representation of the verification code to the communications system, which may then compare the decoded verification code to one or more stored verification codes. If a value of the decoded verification code matches a value of a stored verification code, then an operation associated with the stored verification code may be verified and/or completed. In one embodiment, the command processor290a-nforwards the received audio data containing the verification code to a verification service or communications service, which may perform these operations. In an alternative embodiment, the command processor290a-nforwards the audio data to the communications system, and the communications system decodes the audio data. In another embodiment, server120ais bypassed and speech-detection device110sends the audio data directly to the communications system.

FIG. 2Cis a logical block diagram of a communications system294, in accordance with an embodiment. Server120bincludes the communication system294. The communications system294is a system that is configured to enable voice calls, voice messaging, text messaging and/or other communications between devices. The communications system294includes a verification service295that verifies operations for the communications system. For example, the verification service295may verify phone number pairing operations in which a phone number is added to an account of a user. The verification service295may also verify other contact data before that contact data is associated with a user account. Examples of contact data include a phone number, an email address, an internet protocol (IP) address, a physical address, an additional user account associated with a third party service (e.g., a user account of a social networking service), and so on. In alternative embodiments, the verification service295may be separate from the communications system294, and may verify operations such as adding phone numbers to accounts as well as other operations such as purchase transactions. The verification service295may be included in server120bor in a different server in embodiments.

Server120bmay receive message data143from device115. The message data143may include an indication of contact data to be associated with a user account, such as a phone number, IP address, email address, and so on. Responsive to receipt of the message data143, communications system294may invoke verification service295. Verification service295may include a ticket generator275that generates a new ticket for verification that the contact data should be added to the user account or for verification of some other operation. Generation of the ticket may include adding an entry to a list, table, database, etc. that includes an identification of the user account and the contact data (or operation) to be verified. Code generator271then generates a verification code for the generated ticket as detailed herein. The verification code may be non-audio data, which may be stored in the ticket. Encoder296aencodes the verification code into audio data (e.g., generates an audio representation of the verification code) as detailed herein. Verification service then generates message data149and sends the message data149containing the audio data that represents the verification code or a link to the audio data to device115. The message data149may additionally include an instruction for a user to place device110into a listening mode and/or to issue a verification command to device110.FIG. 5illustrates an example message that has been received by device115. Device115plays the audio data within hearing (pick-up) range of the device110, which has been placed into a listening mode after a user issued a voice command as discussed herein above.

Device110generates audio data293containing a representation of the audio data that was sent to device115in message data149. The audio data293may be received after the communications system294has been granted control of or direct access to device110. Verification service295may include a decoder296bthat processes the audio data293to extract an audio verification code from the audio data. The decoder296bmay use a same audio coding scheme as encoder296athat was used to generate the audio verification code. For example, the decoder296bmay convert frequency information, phase information, etc. from the audio data into numerical and/or alphanumerical values. The decoder296bmay receive the audio data293as an input and output a non-audio data (e.g., a text) representation of the verification code.

Comparator297compares the non-audio representation of the verification code that was extracted from the audio data to one or more stored verification codes (e.g., non-audio data representations of verification codes in one or more tickets). If a match is found between the extracted verification code and a stored verification code, then an associated operation or data (e.g., contact data) may be verified. Action module299may then perform an appropriate action depending on the nature of the operation or data that was verified. For example, if an operation to add a phone number or other contact data to a user account was verified, then the phone number or other contact data may be added to the user account, completing the operation. If a purchase transaction was verified, then a message may be sent to a third party (e.g., a reseller) indicating that the validation was successful. The third party may then complete the purchase transaction.

In one embodiment, audio data293includes audio data representing a digital signature of the verification code. In such an embodiment, the decoder296may extract the digital signature from the audio data in the same manner that the verification code was extracted from the audio data. Signature generator298may then use a stored key to generate a digital signature of the extracted verification code. Comparator297may compare the generated digital signature to a stored digital signature to determine whether the verification code is from a trusted source. If the verification code is from a trusted source and the verification code matches a stored verification code, then action module299may be invoked.

FIG. 2Chas been described for an embodiment in which audio data293was sent directly to server120b. Alternatively, audio data193may be sent to server120aand forwarded to a command processor associated with the communications system. The command processor may include the decoder296, and may extract the verification code from the audio data using the decoder and then send the extracted verification code to server120b. Alternatively, the command processor may forward the audio data to server120bwithout extracting the verification code from the audio data.

The NLU operations described herein may take the form of a multi-domain architecture, such as that illustrated inFIG. 3. In the illustrated architecture, each domain (which may include a set of intents and entity slots that define a larger concept such as music, video, messaging, etc. as well as components such as trained models, etc. used to perform various NLU operations such as NER, IC, or the like) may be constructed separately and made available to the NLU component260during runtime operations where NLU operations are performed on text data (such as text data output from the ASR component250). Each domain may have specially configured components to perform various steps of the NLU operations.

For example, the multi-domain architecture may consist of multiple domains for intents/commands executable by the system100(or by other devices connected to the system100), such as music, video, messaging, and information. The NLU component260may include a plurality of domain recognizers335, where each domain may include its own recognizer263. Each recognizer263may include various NLU components such as an NER component262, IC component264, and other components such as an entity resolver, etc.

For example, a music domain recognizer263-A may have an NER component262-A that identifies what slots (i.e., portions of input text data) may correspond to particular words relevant to the music domain. The slots may correspond to entities such as (for the music domain) a performer, album name, song name, etc. An NER component262may use a machine learning model, such as a domain specific conditional random field (CRF) to both identify the portions of text data corresponding to an entity as well as identify what type of entity corresponds to the text portion. For example, for the text “play songs by the stones,” an NER component262-A trained for a music domain may recognize the portion of text “the stones” corresponds to an entity and an artist name. The music domain recognizer263-A may also have its own IC component264-A that determines the intent of the utterance represented in the text data, assuming that the text data is within the proscribed domain. An IC component264may use a model, such as a domain specific maximum entropy classifier to identify the intent of the utterance, where the intent is the action the user desires the system to perform.

As illustrated inFIG. 3, multiple domains may operate substantially in parallel, with different domain specific components. That is, domain B for verification may have its own recognizer263-B including NER component262-B and IC component264-B. Domain C for messaging may also have similar components in its recognizer263-C, and so on for the different domains available to the system100. When text data300(e.g., ASR output text data) is received, the same text data that is input into the NLU pipeline for domain A263-A may also be input into the NLU pipeline for domain B263-B, where the components for domain B263-B will operate on the text data300as if the text data300related to domain B, the components for domain C263-C will operate on the text data300as if the text data300related to domain C, and so on for the different NLU pipelines for the different domains. Each domain specific NLU pipeline will create its own domain specific NLU results, for example NLU results A (for domain A), NLU results B (for domain B), NLU results C (for domain C), and so on. The different NLU results may then be ranked and further processed using other downstream components as explained below.

As shown inFIG. 3, an NER component262and IC component264may be considered part of a recognizer (such as recognizer263-A,263-B, etc.). The recognizers335may operate using machine learning trained models such as a CRF, maximum entropy classifier, neural network (such as a deep neural network (DNN) or recurrent neural network (RNN), or other classifier. The recognizers335may also use rules that operate on input text data in a particular form to identify named entities and/or intents. The recognizers335may also operate using a data structure such as a finite state transducer (FST) to process the text data300to perform NER and/or IC. Other techniques or models may also be used to perform NER and/or IC. The techniques may also be used together. For example, a set of rules, an FST, and a trained machine learning model may all operate on text data300substantially in parallel to determine the named entities/intents of an utterance represented in the text data300. If one technique performs its task with high enough confidence, the system may use the output of that technique over the others. The system may also prioritize the results of different techniques in certain circumstances (for example rules results may be higher priority than FST results, which may be higher priority than model results, or some other configuration). Each recognizer (such as263-A,263-B,263-C, etc.) may have its own rules, FSTs, and/or models operating such that each recognizer operates substantially in parallel to the other recognizers to come up with its own interpretation of the utterance represented in the text data300.

The output of each recognizer may be an N-best list of intents and slots representing the particular recognizer's top choices as to the meaning of the utterance represented in the text data300, along with scores for each item in the N-best list. Each recognizer of the recognizers335may operate on the text data300substantially in parallel, resulting in a number of different N-best lists, one for each domain (e.g., one N-best list for music, one N-best list for video, etc.). The size of any particular N-best list output from a particular recognizer is configurable and may be different across domains.

While the recognizers335perform NER (e.g., identify words of the input text data300that are important for downstream processing (sometimes called light slot filling), and may even label those words according to type (such as artist name, album name, city, or the like)), the recognizers335may not perform entity resolution (i.e., determining the actual entity corresponding to the words of the text data300). Entity resolution is typically a higher latency process and involves communications with a knowledge base272or other component to precisely identify the specific entities. As this process is resource intensive, it may be preferable to not perform this task for each item of every N-best list across the multiple domains as some items have low scores and are unlikely to be used and any resources spent performing entity resolution would be wasted on low scoring items. Thus, a filtering of potential results may first be performed before engaging in more resource intensive processing. To that end, the cumulative N-best lists340of all the domains may be passed to a cross domain processing component355, which may further rank the individual items in the N-best lists340as well as perform other operations.

The cross domain processing component355may include a cross-domain ranker350. The cross-domain ranker350takes the N-best lists340and selects from among the lists the top choices to create a new N-best list that may include items from different domains, but only includes the highest scoring ones of those domains. The purpose of the cross-domain ranker350is to create a new list of top scoring potential results, so that downstream (more resource intensive) processes may only operate on the top choices.

As an example of a multi-domain N-best list created by the cross-domain ranker350, take the example text data300of “play the hunger games.” The text data300may be processed by each of the recognizers335, and each will output an N-best list, resulting in the group of N-best lists340input into the cross domain processing component355. The cross-domain ranker350may then rank the individual items among the N-best lists to create a new N-best list. Each item in the cross-domain N-best list360may also include a score. The size of the cross domain N-best list360is configurable. While the cross-domain ranker350takes as input the N-best lists340, it may also consider other information, such as other data391.

The cross domain processing component355may also include a light slot filler component352. The light slot filler component352can take text from slots and alter it to make the text more easily processed by downstream components. The operations of the light slot filler component352are typically low latency operations that do not involve heavy operations, such as those that require referencing a knowledge base. The purpose of the light slot filler component352is to replace words with other words or values that may be more easily understood by downstream components. For example, if the text data300includes the word “tomorrow,” the light slot filler component352may replace the word “tomorrow” with an actual date for purposes of downstream processing. Similarly, a word “CD” may be replaced by a word “album.” The replaced words are then included in the cross domain N-best list360.

The cross-domain N-best list360is output to a heavy slot filler and entity resolver370. This component370can apply rules or other instructions to standardize labels or tokens from previous stages into an intent/slot representation. The precise transformation may depend on the domain (for example, for a travel domain a text mention of “Boston airport” may be transformed to the standard BOS three-letter code referring to the airport). The entity resolver370can refer to an authority source (such as a knowledge base272) that is used to specifically identify the precise entity referred to in the entity mention identified in the incoming text. Specific intent/slot combinations may also be tied to a particular source, which may then be used to resolve the text (such as by providing information or a command to be executed in response to a user utterance). In the example “verify operation for John Smith,” the entity resolver370may reference to a personal list of operations and/or data to be verified, or the like. The output from the entity resolver370may include an altered N-best list that is based on the cross-domain N-best list360but also includes more detailed information about the specific entities mentioned in the text (such as specific entity IDs) and/or more detailed slot data that can eventually be used by a command processor290which may be incorporated into the same system components or pipeline or may be on a separate device in communication with the system. While illustrated as a cross-domain resolver, multiple entity resolvers370may exist where a particular entity resolver370may be specific for one or more domains.

As can be appreciated, the entity resolver370may not necessarily be successful in resolving every entity and filling every slot. This may result in incomplete results in the combined N-best list. A final ranker390may consider such errors when determining how to rank the ultimate results for potential execution. For example, if an item of the cross-domain N-best list360comes from a book domain and includes a read book intent, but the entity resolver370cannot find a book with a title matching the input query text, that particular result may be re-scored by the final ranker390to be given a lower score. Each item considered by the final ranker390may also be assigned a particular confidence, where the confidence may be determined by a recognizer263, cross domain processing component355, or by the final ranker390itself. Those confidence scores may be used to determine how to rank the individual NLU results represented in the N-best lists. The confidence scores may be affected by unfilled slots. For example, if one domain is capable of filling a slot (i.e., resolving the word in the slot to an entity or other recognizable form) for an input utterance, the results from that domain may have a higher confidence than those from a different domain that is not capable of filling a slot.

The final ranker390may be configured to apply re-scoring, biasing, or other techniques to obtain the most preferred ultimate result. To do so, the final ranker390may consider not only the NLU results of the N-best lists, but may also consider other data391. This other data391may include a variety of information. For example, the other data391may include application rating or popularity data. For example, if one application has a particularly high rating, the final ranker390may increase the score of results associated with that particular application. The other data391may also include information about applications that have been specifically enabled by the user (as indicated in a user profile204, discussed in reference toFIG. 2A). NLU results from enabled applications may be scored higher than results from non-enabled applications. User history may also be considered, such as if the user regularly uses a particular application or does so at particular times of day. Date, time, location, weather, type of device110, user ID, context, and other information may also be considered. For example, the final ranker390may consider when any particular applications are currently active (such as music being played, a game being played, etc.). The highest scoring result (or results in the case of multiple commands being in an utterance) may be passed to a downstream command processor290for execution.

Following final ranking, the NLU component260may output NLU output data385. The NLU output data385may include an indicator of the intent of the utterance along with data associated with the intent, for example an indication that the intent is “play music” and the music to be played is “Adele.” The NLU output data385may be in the form of previous NLU data such as an item(s) in the N-best lists340, an item(s) in the cross-domain N-best list360, or the like. The NLU output data385may also be in a format executable by the command processor290. Multiple instances of NLU output data (e.g.,385a-385n) may also be output.

Returning toFIG. 2A, a user profile storage204may include data regarding user accounts. The user profile storage204may be located proximate to the server120aand/or to server120b, or may otherwise be in communication with various components, for example over the network(s)199. The user profile storage204may include a variety of information related to individual users, accounts, etc. that interact with the system100. For illustration, the user profile storage204may include data regarding the devices associated with particular individual user accounts, the phone numbers associated with user accounts, voice print identification data (e.g., data that is useable to identify a specific user based on recordings of his or her voice) and so on. User profile storage204may additionally include user defined rules on when to automatically request verification of operations. For example, a user profile may include a rule to automatically perform verification of purchases over $1000. In an example, the user profile storage204is a cloud-based storage. Each user profile may include data such as names of contacts. Each contact may be associated with one or more devices, and each device may be associated with a respective service provider (e.g., a telephony and messaging service provider/carrier). Moreover, each contact device may be associated with data indicating the types of messaging supported by the device. In addition, each service provider/carrier may be associated with data indicating the types of messaging supported by the service provider/carrier.

FIGS. 4A-4Billustrate a sequence diagram showing the verification of an operation (e.g., for linking a phone number to a user account) using audio data. To initiate the sequence, device115sends a message402to server120b, which may be a server for a communications system. In one embodiment, the device115sends the message402to the server120bwithout a user having to input a phone number or other contact information of the server. The user may simply press a verify phone number button in an application on the device115in some embodiments. This may cause the application to call a messaging application on the device (e.g., an SMS application or MMS application) to send the message. Alternatively, this may cause the application to initiate a phone call or video call to a phone number associated with the server120. The message or phone call functions as a request for verification in embodiments. The message may be a text message such as an SMS message or an MMS message. The message or phone call may be sent to a phone number used by the server120bto receive verification requests, the message may be sent to an email address used by the server120bto receive verification requests, or the message may be sent using an API of the server120b. The message may be a synchronous or asynchronous message in accordance with a synchronous communication protocol (e.g., a phone call or video call) or in accordance with an asynchronous communication protocol (e.g., text messaging, audio messaging, voice mail messaging, and so on). The message includes a phone number of the device115in one embodiment, which may be used to respond to the message. If a phone call is made, then the server120bwill be notified of a phone number of the calling device.

Receipt of the message402causes server120bto begin a verification operation for the phone number of the device115. The server120bmay generate an audio verification code at block404. The audio verification code is a unique code generated for a particular verification request in some embodiments. The audio verification code may be a numerical or alphanumeric value that is uniquely generated for the verification of the phone number, and which has been converted to an audio format. To generate the audio verification code, the domain first generates the numerical or alphanumeric code as text. The code may be generated using a random number generator in embodiments. In one embodiment, the generated number is then processed using a cryptographic hash function to produce the verification code. The server120bthen uses an encoder to encode the verification code into audio data (referred to as the audio verification code). The audio data may be an audio file, such as a moving picture experts group audio layer III (MP3) file, a waveform audio (WAV) file, an advanced audio coding (AAC) file, and so on. The server120bmay then invoke a verification service and notify the verification service of the verification code to be used for verification. The verification service may include multiple open verification requests, and may wait to receive the verification code.

In an alternative embodiment, the server120bcalls the verification service, and the verification service opens a new verification request and generates an audio verification code at block404. The verification service then uses an encoder to encode the verification code into audio data (referred to as the audio verification code). The verification service may then provide the audio verification code to, for example, a messaging domain or a verification domain for transmission to a device.

The server120bthen sends a response message406that includes the audio verification code or a link to the audio verification code back to the device. The server120bmay generate a response message that includes the audio verification code, and send the response message back to the phone number of the device using the same messaging protocol that was used to receive the message from the device402. For example, the response message that includes the audio verification code may be an SMS message or MMS message. In some instances, the response message includes a link to the audio verification code rather than an audio file with the audio verification code. The device115may access this link to receive the audio verification code.

The response message may additionally include instructions to say a particular phrase to place speech-detection device110into a listening mode and/or to provide a specific oral command to the speech-detection device110. For example, the response message may instruct the user to say “Device X, verify phone number.” The user of device115may speak the particular phrase, which the speech-detection device110receives at block408via one or more mics. This causes the device110to enter a listening mode. Additionally, the phrase may be captured by device110and captured audio412(e.g., audio data comprising an audio recording or stream containing an utterance of the phrase) may be sent to server120afor analysis112. Server120amay be a server for a speech processing system. The server120amay process the captured audio to identify a particular domain or service to activate (e.g., a verification domain or service).

In one embodiment, the captured audio is processed by user recognition component295to determine an identity of the speaker. Once the speaker is identified, a speaker ID is determined.

In one embodiment, at block414the server performs ASR on the captured audio to generate text data (e.g., to generate text data comprising the text, “Device X, verify phone number.” At block416, the server120aperforms NLU on the text data to determine NLU results. The server120amay determine (418), based on the NLU results, an intent to verify a phone number or another operation. For example, for text data corresponding to “verify phone number,” the server120amay determine the utterance corresponds to a “verify phone number intent.” The NLU may pass the verify phone number intent to a verification domain or a messaging domain (e.g., if the phone number is to be verified for the messaging domain). This intent may be passed to the appropriate domain via notice419along with additional metadata such as a device ID of device110, the device ID of device115, a speaker ID of a speaker who spoke the voice command, and so on.

The device110and device115may be associated with the same user account on server120aand/or sever120b. Accordingly, server120a(e.g., the verification domain or messaging domain) may determine that the earlier request message402that was received from device115to verify a phone number is related to the verify phone number intent. Server120amay determine, therefore, the particular phone number to be verified. Server120amay determine that a verification code was generated for the phone number verification, and that an audio verification code is expected. Alternatively, such determinations may not be made by server120b.

Server120bmay generate at block420a text instruction to play the audio verification code and send the text instruction421to server120a. Server120amay then perform text to speech processing to convert the text instruction into an audio instruction (e.g., a spoken word instruction) at block422. The server120athen sends the audio instruction423to device110, which outputs the audio instruction. For example, the device110may state, “please play the audio verification code.” Subsequently, device115plays the audio verification code424within pickup range of the device110. For example, a user may click on or select an audio file comprising the audio verification code or a link to the audio verification code that was included in the response message to the device115. This may cause the device115to play the audio verification code424.

The played audio verification code is received by one or more mics on device110and captured. The captured audio425of the audio verification code is sent from the device110to the server120b. In one embodiment, while the device110is in a listening mode all audio captured by the device110is streamed from device110to server120b.

Server120bmay decode a portion of the captured audio corresponding to the audio verification code (block428). Server120bmay provide this portion of the captured audio to a decoder, which extracts the verification code from the portion of the captured audio (block430).

In one embodiment, a command processor of the server120ainvokes a callback function to a verification service in server120band provides the verification code to the verification service. The verification service may include multiple open verification requests also referred to herein as tickets, and may determine which open verification request is associated with the verification code. In one embodiment, the callback function is for the specific verification code, and calls the open verification request associated with the verification code. Upon receiving the verification code for the open verification request, the verification service compares the extracted audio verification code to the stored audio verification code to determine whether the two audio verification codes match (block432). Responsive to determining that the audio verification codes match, the verification service verifies the phone number or verifies the requested operation (block434). This may indicate that the requested verification is complete, and that verification was successful. The server120b(e.g., the verification domain or messaging domain) may then proceed with a requested operation, such as linking a particular phone number to a user account. The operation may have been initiated by a user, and the verification may indicate that the user had authorization to perform or complete the operation. Verification of a phone number or other contact data may include linking the phone number or other contact data to the user account associated with the device115and/or the device110. This may enable the device115and/or device110to perform one or more operations using the phone number. For example, this may enable the device110to make or receive VOIP phone calls to the phone number via a messaging domain of server120b.

As noted, the device115and the device110may both be associated with a user account. Alternatively, just the device110may be registered to the user account or just the device115may be registered to the user account. The fact that the device115was able to receive the audio verification code at a particular phone number, and that the device110was able to also receive the audio verification code played by device115, and provide it back to server120b, confirms to server120bthat the user has access to both device110and device115. This additionally confirms that the user has access to the phone number that he or she is claiming ownership of. This therefore confirms that the person who claimed to have a particular phone number actually has that phone number. This prevents users from falsely claiming phone numbers not belonging to them.

The server120amay also identify a user that spoke the utterance (provided the voice command). Identifying the user may involve processes detailed with respect to the user recognition component295discussed above. Identifying the user may also include the system prompting the user to speak user identifying information, such as the user's name, user's system unique ID, etc. This may be another level of verification.

The above described flow may be reversed such that the audio verification code is sent to device110, played by device110and recorded by device115. For example, a user may issue a command of “Device, verify operation” to device110. A recording of this voice command may be sent to server120aand processed by the ASR and NLU to determine that the user intent is to verify an operation. This intent may be passed on to a verification domain or messaging domain, for example. The verification domain or messaging domain would call the verification service to open a verification request. The verification service or appropriate domain would generate the audio verification code and provide it to device110. The user would place the device115into a listening mode, and the device110would play the audio verification code. The device115may capture the audio verification code and then provide the captured audio of the audio verification code to server120aor server120bfor verification. The server may extract the verification code and execute a callback function to the verification service. The verification service may verify that the extracted verification code matches a stored verification code and then execute a callback function to the appropriate domain.

FIG. 5illustrates an example of a text message received by a mobile phone after requesting verification of a phone number. The device115may include a display103that presents message payloads as well as other content. The display103may present a SMS message or other message including message content502in the form of text. The received text message includes message content502with instructions to place a speech-detection device into listening mode. The message may include text stating, for example, “say the following to your speech-detection device, “device X, verify phone number.” In one embodiment, the message content502includes audio that comprises a wake word and/or instructions. The text message additionally includes a link504to message content audio data, which may be displayed. The link504may be text, an icon, or any other type of visual data that may operate as a link. The message content audio data may be audio data comprising an audio verification code. The message content may additionally include audio comprising a wake word that will place the speech-detection device into a listening mode. The wake word may precede the audio verification code in the audio data. In one embodiment, the audio data may is in the message. In such an embodiment, the message may be, for example, an MMS message.

Embodiments have been discussed above with reference to verifying a phone number to link that phone number to a user account. However, the methods, techniques and systems described wherein with reference to phone number verification can also be used to verify other types of information, operations and transactions. For example, audio data may be used to verify purchases, to verify that a person has access to an area or to specific data, and so on.

FIGS. 6-8are flow diagrams illustrating methods for verifying an operation using an audio verification code according to embodiments of the present disclosure. The methods may be performed by a processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, at least some operations of the methods are performed by server(s)120a-bofFIG. 1.

For simplicity of explanation, the methods are depicted and described as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts may be required to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states via a state diagram or events.

FIG. 6is a flow diagram illustrating a method600for verifying an operation using an audio verification code according to embodiments of the present disclosure. The operation may have been initiated by a user who may or may not have authorization to perform the operation. At block605of method600, processing logic receives a request to verify an operation. The request may be received as first message data, which may represent a request. This request may be received from a third party server or from an associated server of processing logic. For example, third parties may request verification of purchase transactions that have been generated on websites of those third parties. In one embodiment, a child may go shopping, and that child may have access to a parent's user account. That child may make a purchase under the user account. The purchase may not be confirmed, however, until a verification process is completed for that purchase, as described herein. The request may also be received from a user who wishes to access data or to perform some restricted action. The request may also be a request to link a phone number to a user account (to verify a phone number). In such an embodiment, the request may include a phone number to verify. This phone number may be included in the request message (e.g., if the request message is received via an SMS or MMS message from the phone number). The phone number may also be determined by a device115using an API for determining the device's phone number. The request may be received via an email, a text message, an instant message, an API call, or any other type of message.

At block610, processing logic (e.g., a domain, skill or service) generates a verification code for the operation. The verification code may be generated using a random number generator, pseudorandom number generator, or other type of code generator. The verification code may be a numerical value, an alphanumeric value (e.g., a hexadecimal value), or other text based or binary value. In some embodiments, a random value is first generated, which is then processed using a cryptographic hash function to result in the verification code. The verification code may be generated in the form of a first text version of the verification code, which may be stored for later comparison.

At block615, the verification code is encoded into audio data. This may include generating first audio data representing the verification code and/or adding the first audio data to an existing audio file. The audio data may contain just an audio version of the verification code or an audio version of the verification code along with other audio data. For example, the audio verification code may in the ultrasonic frequency range (e.g., above 20,000 Hz) and the audio data may also include additional audio at frequencies audible to the human ear (e.g., audible frequency range of 20-20,000 Hz). The additional audio may include spoken instructions or information, a song, a pleasant tone, and so on. The audio verification code and the additional audio may be superimposed in the audio data so that they will play at the same time.

At block620, message data comprising the audio data or a link to the audio data is sent to a first device. The first device may be a mobile device such as a mobile phone, a tablet computer, a laptop computer, and so on. The first device may also be a traditionally immobile device such as a desktop computer. The first device may also be a speech-detection device, such as an Amazon Echo, a Google Home, an Apple HomePod, etc. The message data may be sent to an email address, to a phone number, to a particular device ID, and so on. The audio data may be a file attached to the message or may be a link (e.g., a URL) in the message that points to the audio data. If the message data is sent to an email address, then the first device may have access to the email address and may download the message from the email address.

At block625, processing logic receives from a second device second audio data representing the first audio data. This may be an audio recording or stream that was generated by the second device. The second device may be a mobile device such as a mobile phone, a tablet computer, a laptop computer, and so on having a speech-detection application installed thereon. The second device may also be a traditionally immobile device such as a desktop computer having a speech-detection application installed thereon. The second device may also be a speech-detection device, such as an Amazon Echo, a Google Home, an Apple HomePod, etc. The second message data may be generated by the second device when the first device plays the audio data. Accordingly, the second message data will comprise a representation of the first audio data. The second device may be a speech-detection device or a device that has a speech-detection application installed thereon. The speech-detection device or application may capture the playback of the first audio data and send a recording or stream to a server comprising the processing logic.

At block630, processing logic extracts the verification code from the audio recording or stream. This may include processing the second audio data using a decoder to generate a second text version of the verification code. At block635, processing logic determines that the extracted verification code matches the verification code that was generated for the operation and verifies the operation. In one embodiment, the extracted verification code (second text version of the audio verification code) is compared against a particular verification code (first text version of audio verification code), or some or all verification codes that have been generated and that have not yet been matched to an extracted verification code. A match may be found when a value of the first text version of the audio verification code is equal to a value of the second text version of the audio verification code. When a matching verification code is identified, a user account associated with the identified verification code is determined, one or more verification conditions associated with that user account may then be determined. Such verification conditions may include a user verification condition, a device verification condition, a phone number verification condition, a location verification condition, and so on.

For a device verification condition, the second device may be configured as an approved device that is authorized to verify operations for a particular user account. The second device may be linked to the user account, and the second device may be identified in the user account as such an authorized device. The second device may have a unique device ID or a unique application ID for an application installed on the second device. For example, the second device may be a speech-detection device having a fixed approved location in a user's home. Accordingly, any audio verification codes received from the second device may be assumed to be approved by the user associated with the user account since presumably only that user or someone authorized by that user would have access to the second device.

For a user verification condition, voice data may be received from a user of the second device and used to verify an identity of the user using voice print identification and/or successful answers to one or more questions asked to verify the user's identity. If the user is listed in the user account as authorized to verify operations, then verification may be performed.

For a location verification condition, information from the second device may be used to verify a location of the second device. For example, the second device may include a global positioning system (GPS) receiver, which may be used to determine a location of the second device. Alternatively, or additionally, the second device may use one or more wireless communication technologies to communicate with one or more cell towers or Wi-Fi access points. Processing logic may use triangulation to determine the location of the second device. Alternatively, or additionally, the second device may be on a local area network (LAN) having a particular internet protocol (IP) address, and the second device may be associated with that IP address. The IP address may be used to determine the location of the second device (e.g., the IP address may be listed as the home IP address for a user account).

For a phone number verification condition, the audio data comprising the audio verification code may be sent to a particular phone number that is listed as an approved phone number for verification purposes in the user account. If the audio verification code is then received by the second device, this may confirm that the owner of the phone number received and approved the operation to be verified.

The operation to be verified may be an operation that was initiated by a user who may or may not have authorization to perform the operation. The verification may indicate that the user who initiated the authorization did have authorization to perform or complete the operation, or that a second user authorizes the operation after the operation was initiated. Verification of an operation may include sending a message to a domain or application notifying the domain or application that the operation was successfully verified. In the example of a purchase transaction, processing logic may determine that the purchase is approved and send a notice to a requesting system indicating that the purchase is approved. Example operations include operations to add data to an account (e.g., to associate a phone number with a user account) and transactions associated with a user account (e.g., a purchase transaction associated with a user account). In the case of a purchase transaction initiated on a third party system, processing logic may sent a message to the third party system confirming that the entity that initiated the purchase transaction has authorization to complete the purchase transaction, or that the purchase transaction is otherwise to be completed.

FIG. 7is a flow diagram illustrating an additional method700for verifying an operation using an audio verification code according to embodiments of the present disclosure. At block705of method700, processing logic receives a notice of a verification request. The notice may include a nonce, which may be a unique one time value associated with the verification request. The notice may be received from an application running on a device (e.g., on a mobile phone of a user). At block710, processing logic receives a request to verify an operation. The request may include a second nonce.

At block715, processing logic compares the first nonce to the second nonce. If the two nonces match, then the method may continue to block725. If the nonces do not match, then the method proceeds to block720and processing logic determines that the request is an invalid request. This may ensure that only authorized applications or systems are able to request verification. These operations may be performed, for example, by a verification domain or communication domain as described herein.

At block725, processing logic determines that the request is a valid request. At block730, processing logic generates a verification code for the verification request. At block735, processing logic generates a digital signature of the verification code using a key (which may be a private key). The digital signature may be used to verify that a verification code is not modified by a recipient. In one embodiment, the digital signature is a cryptographic hash of the verification code. The digital signature may be, for example, a message authentication code (MAC) such as a keyed-hash message authentication code HMAC. To generate the digital signature of the verification code, processing logic may use a key generation algorithm to select a key from a key space. The key may be chosen from the key space at random. The key space represents all the possible permutations for keys. The key space may be a predefined space, such as all 8 digit integers, for example. In another example, the key space may be 256 bits (thus containing 2{circumflex over ( )}256 or 71.579e77 possible keys) or another number of bits. Once the key is generated, a cryptographic hash function may be applied to the verification code along with the key. This may generate a cryptographic hash that acts as a digital signature. This digital signature may later be used to verify the verification code by comparing the digital signature with a new digital signature generated from the verification code and the key using the cryptographic hash function. At block740, processing logic encodes the verification code and the digital signature into audio data. This may include generating first audio data that represents the verification code and third audio data that represents the digital signature using an encoder.

At block745, processing logic sends a message comprising the audio data (or a link to the audio data) to a first device (e.g., to a mobile phone, tablet computer, speech-detection device, etc.). At block750, processing logic receives, from a second device (e.g., to a mobile phone, tablet computer, speech-detection device, etc.), captured audio (e.g., an audio recording or stream) generated by the second device. The captured audio may be second audio data that represents the first audio data and the third audio data. The captured audio therefore includes the first audio data and third audio data, which was played by the first device in the vicinity of the second device. At block755, processing logic extracts the verification code and the digital signature from the captured audio using a decoder that corresponds to the encoder that was used to encode the digital signature and verification code into an audio format. At block760, processing logic generates a new digital signature of the extracted verification code using the key.

At block765, processing logic compares the generated digital signature to the extracted digital signature. This operation may be performed, for example, by a verification service that is called by a verification domain or messaging domain. If the generated digital signature that was generated from the extracted verification code matches the extracted digital signature, then processing logic determines that the verification code is validated. Processing logic then proceeds to block770, and the operation is verified. If the generated digital signature fails to match the extracted digital signature, then the method proceeds to block775, and the operation is not verified.

FIG. 8is a flow diagram illustrating an additional method800for verifying an operation using an audio verification code according to embodiments of the present disclosure. At block805of method800, processing logic receives a request to verify an operation. Processing logic may open a verification request in response. At block810, processing logic generates a verification code for the operation. The verification code may be generated by an appropriate domain or by a verification service that is called by a domain. At block815, processing logic encodes the verification code into audio data. At block820, processing logic sends a message comprising the audio data (or a link to the audio data) to a first device.

A user causes the first device to play the audio data in the vicinity of a second device, which records the playback of the audio data. Processing logic then receives, from the second device, captured audio generated by the second device, where the captured audio comprises the audio data and potentially an utterance of a speaker. The speaker may have given a voice command to the second device prior to playing the audio data on the first device. The command may have been a wake command and/or a verification command, for example.

At block830, processing logic determines an identity of the speaker who activated the second device from the captured audio. Alternatively, the identity of the speaker may be determined by generating textual questions, converting the textual questions to audio files containing spoken questions, providing the audio files to the second device, receiving additional recordings comprising voice responses of the user, converting the voice responses to text responses, and identifying the user ID from the text responses. At block835, processing logic determines whether the user is authorized to perform the verify operation. For example, a user profile may include one or more verification rules, which may specify a specific user who is able to perform verifications, a particular device that is able to perform verifications, and so on. If the user is authorized to perform verifications, the method continues to block840. Otherwise, the method proceeds to block860and the operation is not verified.

At block840, processing logic may determine whether the second device is authorized to perform the verify operation. This may be determined by comparing a device ID of the second device to a verification rule in the user profile. If the second device is authorized to perform the verify operation, the method continues to block845. Otherwise the method continues to block860.

At block845, processing logic extracts the verification code from the captured audio. At block850, processing logic determines whether the extracted verification code matches a stored verification code (e.g., associated with an open verification request). If so, the method continues to block855, and the operation is verified. Otherwise, the method proceeds to block860, and the operation is not verified.

FIG. 9is a block diagram conceptually illustrating a user device, such as the device110and/or device115, that may be used with the described system100.FIG. 10is a block diagram conceptually illustrating example components of a remote device, such as the server(s)120, which may assist with ASR processing, NLU processing, or command processing. Multiple servers120a-bmay be included in the system100, such as one server for performing ASR, one server for performing NLU, etc. In operation, each of these devices (or groups of devices) may include computer-readable and computer-executable instructions that reside on the respective device110,115,120a-b, as will be discussed further below.

Each of these devices110,115,120a-bmay include one or more controllers/processors904,1004, which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory906,1006for storing data and instructions of the respective device. The memories906,1006may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive (MRAM), and/or other types of memory. Each device110,115,120a-bmay also include a data storage component908,1008for storing data and controller/processor-executable instructions. Each data storage component908,1008may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device110,115,120a-bmay also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces902,1002.

Computer instructions for operating each device110,115,120a-band its various components may be executed by the respective device's controller(s)/processor(s)904,1004, using the memory906,1006as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory906,1006, storage908,1008, or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.

Each device110,115,120a-bincludes input/output device interfaces902,1002. A variety of components may be connected through the input/output device interfaces902,1002, as will be discussed further below. Additionally, each device110,115,120a-bmay include an address/data bus924,1024for conveying data among components of the respective device. Each component within a device110,115,120a-bmay also be directly connected to other components in addition to (or instead of) being connected to other components across the bus924,1024.

Referring toFIG. 9, the device110,115may include input/output device interfaces902that connect to a variety of components such as an audio output component such as a speaker(s)918, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device110,115may also include an audio capture component. The audio capture component may be, for example, a microphone920or array of microphones, a wired headset or a wireless headset (not illustrated), etc. The microphone(s)920may be configured to capture audio. If an array of microphones918is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array.

Via antenna(s)914, the input/output device interfaces902may connect to one or more networks199via a wireless local area network (WLAN) (such as Wi-Fi) radio, Bluetooth, and/or wireless network radio, such as a radio capable of communication with a wireless communication network such as a Long Term Evolution (LTE) network, WiMAX network, 3G network, 4G network, 5G network, etc. A wired connection such as Ethernet may also be supported. Through the network(s)199, the system100may be distributed across a networked environment.

As noted above, multiple devices may be employed in a single speech processing system100. In such a multi-device system, each of the devices may include different components for performing different aspects of the speech processing. The multiple devices may include overlapping components. The components of the device(s)110,115and the server(s)120, as illustrated inFIGS. 9 and 10, are exemplary, and may be located as a stand-alone device or may be included, in whole or in part, as a component of a larger device or system.

As illustrated inFIG. 11, multiple devices (110a-110e,120,125) may contain components of the system100and the devices may be connected over a network(s)199. The network(s)199may include a local or private network or may include a wide network such as the Internet. Devices may be connected to the network(s)199through either wired or wireless connections. For example, a speech-detection device110a, a smart phone110b, a smart watch110c, a tablet computer110d, and/or a vehicle110emay be connected to the network(s)199through a wireless service provider, over a Wi-Fi or cellular network connection, or the like. Other devices are included as network-connected support devices, such as the server(s)120, the service provider server(s)125, or others. The support devices may connect to the network(s)199through a wired connection or wireless connection. Networked devices may capture audio using one-or-more built-in or connected microphones or audio capture devices, with processing performed by ASR, NLU, or other components of the same device or another device connected via the network(s)199, such as the ASR component250, the NLU component260, etc. of one or more servers120.