Voice user interface entity resolution

Techniques for resolving ambiguous recipient users and/or recipient devices are described. A system receives input data from a device. The system may determine the input data corresponds to a spoken utterance requesting a call be established. When the system determines the recipient user and/or the recipient device is ambiguous, the system determines communications pattern data associated with the originating device. The system also determines context data. Based on the communications pattern data and the context data, the system may resolve an identity of the recipient user and/or resolve an identity of the recipient device. The system may also use communications pattern data and context data to determine when a device is likely to receive a communication and output content indicating same to a user.

BACKGROUND

Speech recognition systems have progressed to the point where humans can interact with computing devices using their voices. Such systems employ techniques to identify the words spoken by a human user based on the various qualities of a received audio input. Speech recognition combined with natural language understanding processing techniques enable speech-based user control of a computing device to perform tasks based on the user's spoken commands. The combination of speech recognition and natural language understanding processing techniques is referred to herein as speech processing. Speech processing may also involve converting a user's speech into text data which may then be provided to various text-based software applications.

Speech processing may be used by computers, hand-held devices, telephone computer systems, kiosks, and a wide variety of other devices to improve human-computer interactions.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics concerned with transforming audio data associated with speech into text data representative of that speech. Natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics concerned with enabling computers to derive meaning from text input containing natural language. Text-to-speech (TTS) is a field of computer science, artificial intelligence, and linguistics concerned with enabling computers to output synthesized speech. ASR, NLU, and TTS may be used together as part of a speech processing system.

A speech controlled computing system may answer user commands requesting the output of content. For example, a user may say “Alexa, what is the weather.” In response, the system may output weather information. For further example, a user may say “Alexa, play Adele.” In response, the system may output music sang by Adele. Yet further, for example, a user may say “Alexa, call John Smith.” In response, the system may establish a call between the user's device and a device associated with John Smith.

A speech controlled computing system may also provide communications functionality (e.g., calls or messaging) between two or more users. For example, a user may say “Alexa, call Joe.” If a contact list of the user includes multiple “Joes,” the system may output computerized speech corresponding to “which Joe would you like to call?” If the contact list of the user only includes one “Joe,” the system may output computerized speech corresponding to “Joe Smith, correct?” When the user responds to the computerized speech by either saying the full name of the Joe the user wants to call or indicates the individual represented in the computerized speech is correct, depending on the situation, the system establishes a call between the user's device and a device of the recipient individual.

The present disclosure improves speech controlled computing systems by enabling the system to resolve the recipient of a call or message without needing to have the originating user confirm the recipient's identity. The system analyzes previous instances of system communications that a user was involved in to determine communication patterns. Such patterns may recur on a specific day of a week, a specific day of a month, a specific day of a year, etc.

An originating user may request the system make a call to a recipient user, for example by speaking “Alexa, call Joe” to a device. The system may use pre-determined pattern information and other present context information to resolve an identity of the recipient user (e.g., Joe). For example, the system may determine it is about 6:00 pm on a Sunday for a particular user. The system may also determine that the originating user's previous interaction history represents a pattern where the originating user routinely calls Joe Smith on Sundays around 6:00 pm. The system may thus determine the recipient user is Joe Smith and may establish a call between the originating user's device and Joe Smith's device without needing to ask the originating user to confirm s/he wants to call Joe Smith.

As described, the current context information used by the system to determine an applicable pattern may include time information, date information as well as other information. For example, the current context information may also include location information. For example, a pattern may indicate that the originating user routinely calls Joe Smith at about 6:00 pm on Sundays, but only using a device located in a kitchen of the originating user's house. The current context information may also include presence information. For example, a pattern may indicate that the originating user routinely calls Joe Smith at about 6:00 pm on Sundays, but only when three individuals (e.g., the originating user and his/her family) are present. The current context information may include other kinds of information as described in detail below.

FIGS. 1A and 1Billustrate a system in which devices (110a/110b) local to respective users (5/7), one or more server(s)120, and one or more communications servers125may communicate across one or more networks199. Although the figures and discussion illustrate certain operational steps of the system in a particular order, the steps described may be performed in a different order (as well as certain steps removed or added) without departing from the intent of the disclosure.

FIG. 1Aillustrates a system configured to resolve an identity of a recipient user. An originating user5speaks a command requesting a communication be established (e.g., a two-way call) or performed (e.g., sending of a message), such as “Alexa, call Joe” or “Alexa, send Joe a message.” One or more microphones of an originating device110acapture input audio11corresponding to the spoken command. The originating device110agenerates input audio data corresponding to the input audio11and sends the input audio data to the server(s)120.

The server(s)120receives the input audio data and performs (130) ASR on the input audio data to generate input text data. The server(s)120performs (132) NLU on the input text data to determine a spoken command represented in the input text data corresponds to a <Call> intent as well as determine a potential recipient user. For example, if the spoken command corresponds to “Alexa, call Joe,” the server(s)120may determine “Joe” as being the potential recipient user. At this point, the server(s)120has not made a determination regarding the identity of “Joe.”

The server(s)120determines (136) current context data associated with the originating user5. The current context data may include information representing a geographic location of the originating device110a, time information, date information, originating user5information, and/or information indicating a number of individuals detectable by the originating device110a. The aforementioned list of information represented in the current context data is not exhaustive. Thus, it should be appreciated that the current context data may include additional or other information than what is explicitly detailed above.

The server(s)120determines communication history data associated with the originating user5, and determines (138) at least one pattern represented in the communication history data. For example, a pattern may correspond to the originating user5calling a particular recipient user every day at about 5:00 pm. For further example, a pattern may correspond to the originating user5calling a particular recipient user every Sunday at about 6:00 pm using an originating device110alocated in the originating user's kitchen. Other patterns are also possible.

The server(s)120determines (140) one of the determined at least one patterns that is applicable to the current communication based on the current context data. As described, each pattern may be associated with a particular recipient user. The server(s)120may resolve (142) an identity of the potential recipient user based on the determined pattern. For example, if the potential recipient user corresponds to “Joe” and the pattern is associated with “Joe Smith,” the server(s)120may resolve the potential recipient user7to be “Joe Smith.”

The server(s)120determines (144) a recipient device110bassociated with the recipient user7. The recipient device110bmay communicate directly with the server(s)120via the network(s)199. Alternatively, the recipient device110bmay communication with a telephone network, which communicates with the server(s)120via the network(s)199. The server(s)120then causes (146) communication to occur between the originating user5and the recipient user7. The communication between the originating user5and the recipient user7may be facilitated by a communications server(s)125. For example, if the user spoke “Alexa, call Joe,” the server(s)120may establish a two-way communication channel between the originating device110aand the recipient device110b. A two-way communication channel may correspond to a phone call, Voice Over Internet Protocol (VOIP) call, etc. For further example, if the user spoke “Alexa, send Joe a message,” the server(s)120may send output audio data corresponding to message content to the recipient device110b.

FIG. 1Billustrates a system configured to output a prompt to a recipient user7. The server(s)120determines (148) at least one pattern represented in communication history data associated with the recipient user7. The server(s)120also determines (150) current context data associated with the recipient user7. The server(s)120determines (152) one of the determined at least one patterns based on the current context data.

The system may proactively prompt the recipient user7with respect to future potential communications. The server(s) may generate (154) output text data representing the determined pattern. For example, the output text data may correspond to “John may be calling you in 5 minutes based on your previous calls with John.” The server(s)120may cause text represented in the output text data to be presented on a display of the recipient device110b. Alternatively, the server(s)120may perform (156) TTS on the output text data to generate output audio data and may cause (158) the recipient device110bto output audio corresponding to the output audio data.

The system may operate using various components as described inFIG. 2. The various components may be located on a same or different physical devices. Communication between various components may occur directly or across a network(s)199.

The originating device110amay receive input audio11, corresponding to a spoken utterance corresponding at least in part to a command, using an audio capture component, such as a microphone or array of microphones. The originating device110a, using a wakeword detection component220, processes audio data corresponding to the input audio11to determine if a keyword (e.g., a wakeword) is detected in the audio data. Following detection of a wakeword, the originating device110asends input audio data211, corresponding to the utterance, to the server(s)120.

Upon receipt by the server(s)120, the input audio data211may be sent to an orchestrator component230. The orchestrator component230may include memory and logic that enables the orchestrator component230to transmit various pieces and forms of data to various components of the system.

The orchestrator component230sends the input audio data211to a speech processing component240. An ASR component250of the speech processing component240transcribes the input audio data211into input text data representing one more hypotheses representing speech contained in the input audio data211. The ASR component250interprets the utterance in the input audio data211based on a similarity between the utterance and pre-established language models. For example, the ASR component250may compare the input audio data211with models for sounds (e.g., subword units, such as phonemes, etc.) and sequences of sounds to identify words that match the sequence of sounds spoken in the utterance represented in the input audio data211. The ASR component250sends the input text data generated thereby to an NLU component260of the speech processing component260. The input text data sent from the ASR component250to the NLU component260may include a top scoring hypothesis or may include an N-best list including multiple hypotheses. An N-best list may additionally include a score associated with each hypothesis represented therein. Each score may indicate a confidence of ASR processing performed with respect to the hypothesis with which it is associated.

The NLU component260attempts to make a semantic interpretation of the phrases or statements represented in the input text data input therein. That is, the NLU component260determines one or more meanings associated with the phrases or statements represented in the input text data based on individual words represented in the input text data. The NLU component260determines an intent (e.g., an action that a user desires be performed) representing the spoken utterance represented in the input text data as well as pertinent pieces of information in the input text data that allow a device (e.g., the originating device110a, the recipient device110b, the server(s)120, the communications server(s)125, etc.) to execute the intent. For example, if the input text data corresponds to “call mom,” the NLU component260may determine a <Call> intent (e.g., representing a user intended to establish a call) and determine a potential recipient user as “mom.”

The system may include a communications pattern component265that determines patterns in data representing previous system communications. The communications pattern component265takes as input data representing previous system communications (e.g., calls, messages, etc.) of various system users. With respect to each user, the communications pattern component265determines the presence of one or more patterns in the user's previous communications. The patterns may be established based on events occurs on a daily basis, weekly basis, monthly basis, yearly basis, etc.). Moreover, patterns may be established based on events occurring with respect to specific times, locations, etc. The communications pattern component265may be included in the system, for example, as part of server(s)120, and/or the communications pattern component265may be included with and controlled by an application290, such as a communication application associated with communication server(s)125. Different communication applications may include different communication pattern components for purposes of performing the operations described herein differently depending on the different communication application being invoked.

The communications pattern component265may determine patterns based on absolute date. For example, the communications pattern component265may determine a user calls a recipient on August 28thevery year. The communications pattern component265may also determine patterns based on non-absolute data. For example, the communications pattern component265may determine a user calls a recipient every Sunday, a user calls a recipient every other Thursday, etc. The communications patterns component265may also determine patterns based on time. For example, the communications patterns component265may determine a user calls a recipient every Sunday at about 6:00 pm. The communications pattern component265may also determine patterns based on presence data. For example, the communications pattern component265may determine a user calls a recipient every Sunday at about 6:00 pm when the user's family is present with the user (e.g., the user and his/her family may all engage with the same user device). The communications pattern component265may also determine patterns based on location. For example, the communications pattern component265may determine a user calls a recipient every Sunday at about 6:00 pm from a device110in a kitchen of the user's house. The communications pattern component265may also determine patterns based on content presented on a screen of a device110. For example, an originating user5may routinely call his/her mom when recipe content is presented whereas the originating user5may routinely call his/her dad when sports content is presented. The communications pattern component265may determine other patterns as well.

The communications pattern component265may maintain historical data separate from historical data maintained by the speech processing component240. For example, the communications pattern component265may maintain data relating to previous communications established by the system (for example, for communications specific to a particular application290a) whereas the speech processing component240may maintain data relating to ASR and NLU processing of data.

The communications pattern component265may determine patterns in system communications as the system performs the communications. Alternatively, the communications pattern component265may determine patterns in system communications only when the communications pattern component265receives a request for pattern data604from a communications application290a(described in detail herein below with respect toFIG. 6).

The communications pattern component265may determine a pattern even if each instance of the same communication is not identical. For example, if a user calls Joe Smith on Sunday at 6:01 pm a first time, calls Joe Smith the next Sunday at 6:07 pm, and calls Joe Smith the next Sunday at 5:52 pm, the communications pattern component265may recognize a pattern among these interactions even though they each occurred at a different time. The pattern determined therefrom may be, for example, the user calls Joe Smith every Sunday at about 6:00 pm.

The communications pattern component265may also determine a pattern even if a frequency of the same communications slightly deviates. For example, if a user calls Joe Smith every Sunday, the communications pattern component265may determine the pattern exists even if the user infrequently (e.g., once a month, once every two months, etc.) does not call Joe Smith on a Sunday.

The communications pattern component265may implement decay models that prevent the communications pattern component265from outputting pattern information that is not current. Once a user has engaged in enough ritualistic communications, the communications pattern component265recognizes a pattern corresponding to the ritualistic communications. The user may stop engaging in the ritualistic communications after the communications pattern component265has recognized the pattern. When this occurs, the pattern may be considered to have decayed and the communications pattern component265may no longer consider the pattern a pattern. Alternatively, after the communications pattern component265recognizes the pattern, the user may change the frequency at which the user engages in the ritualistic communications. For example, the pattern may correspond to a frequency of every week, and the user engaged in communications every other week at some point in time after the pattern is recognized. When this occurs, the pattern may be considered to have decayed and the communications pattern component265may no longer consider the pattern a pattern. If the user engages in the communications at the new frequency long enough, the communications pattern component265may recognize a new pattern associated with the new frequency.

In order to apply machine learning techniques, machine learning processes themselves need to be trained. Training a machine learning component, such as the communications pattern component265, requires establishing a “ground truth” for training examples. In machine learning, the term “ground truth” refers to the accuracy of a training set's classification for supervised learning techniques. Various techniques may be used to train the models including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

The server(s)120may include a user recognition component295. The user recognition component295may take as input the input audio data211and/or input text data output by the ASR component250. The user recognition component295determines scores indicating whether input spoken utterance originated from a particular user. For example, a first score may indicate a likelihood that the input spoken utterance originated from a first user, a second score may indicate a likelihood that the input spoken utterance originated from a second user, etc. The user recognition component295also determines an overall confidence regarding the accuracy of user recognition operations. The user recognition component295may perform user recognition by comparing speech characteristics in the input audio data211to stored speech characteristics of users. The user recognition component295may also perform user recognition by comparing biometric data (e.g., fingerprint data, iris data, etc.) received by the system in correlation with the present input utterance to stored biometric data of users. The user recognition component295may further perform user recognition by comparing image data (e.g., including a representation of at least a feature of a user) received by the system in correlation with the present input utterance with stored image data (e.g., including representations of features of users). The user recognition component295may perform additional user recognition processes, including those known in the art. Output of the user recognition component295may be used to inform NLU component processes as well as processing performed by one or more applications290.

The server(s)120may include a user profile storage270. The user profile storage270may include a variety of information related to individual users, groups of users, etc. that interact with the system. The user profile storage270may include one or more customer profiles. Each customer profile may be associated with a different customer identifier (ID). A customer profile may be an umbrella profile specific to a group of users. That is, a customer profile encompasses two or more individual user profiles, each associated with a respective unique user ID. For example, a customer profile may be a household profile that encompasses user profiles associated with multiple users of a single household. A customer profile may include preferences shared by all the user profiles encompassed thereby. Each user profile encompassed under a single customer profile may include preferences specific to the user associated therewith. That is, each user profile may include preferences unique from one or more other user profiles encompassed by the same customer profile. A user profile may be a stand-alone profile or may be encompassed under a customer profile. As illustrated, the user profile storage270is implemented as part of the server(s)120. However, it should be appreciated that the user profile storage270may be located proximate to the server(s)120, or may otherwise be in communication with the server(s)120, for example over the network(s)199.

FIG. 2illustrates various1P applications290(e.g., applications maintained and/or operated by the server(s)120). However, it should be appreciated that the server(s)120may additionally or alternatively communicate with one or more application servers executing3P applications (e.g., applications not maintained and/or operated by the server(s)120, but in communication with the server(s)120).

An “application,” as used herein, may be considered synonymous with a “skill.” A skill may be software akin to an application. That is, a skill may enable the server(s)120or other application server(s) (e.g., the communications server(s)125) to execute specific functionality. The system may be configured with more than one skill. A skill may either be executed by the server(s)120or merely associated with the server(s)120(i.e., one executed by an application server(s)). For example, a weather skill may enable the server(s)120to execute a command with respect to a weather information provider server(s), a ride booking skill may enable the server(s)120to execute a command with respect to a taxi or ride sharing provider server(s), an order pizza skill may enable the server(s)120to execute a command with respect to a restaurant server(s), etc.

The server(s)120may include a TTS component280that generates audio data from text data using one or more different methods. In one method of synthesis called unit selection, the TTS component280matches text data against a database of recorded speech. The TTS component280selects units of recorded speech matching the text data and concatenates the units together to form output audio data. In another method of synthesis called parametric synthesis, the TTS component280varies parameters such as frequency, volume, and noise to create output audio data including an artificial speech waveform. Parametric synthesis uses a computerized voice generator, sometimes called a vocoder.

FIG. 3illustrates the user profile storage270that includes data regarding customer profiles as well as individual user profiles302. Each user profile302may include information indicating various devices. Each user profile302may also include location information associated with each device. Each user profile302may also be associated with pattern data604discussed herein. Each user profile302may additionally include other data not explicitly illustrated.

FIG. 4illustrates how NLU processing is performed on input text data. NLU component260may process text data including several hypotheses of a single utterance. For example, if the ASR component250outputs text data including an N-best list of hypotheses, the NLU component260may process the text data with respect to all (or a portion of) the hypotheses represented therein.

The NLU component260may annotate text represented in text data by parsing and/or tagging the text. For example, for the text “call Joe,” the NLU component260may tag “call Joe” as a command as well as tag “Joe” as an entity corresponding to a recipient of the call.

The NLU component260may include one or more recognizers463. Each recognizer463may be associated with a different application290. Each recognizer463may process with respect to text data input to the NLU component260. For example, each recognizer463may operate in parallel with other recognizers463of the NLU component260.

Each recognizer463may include a named entity recognition (NER) component462. The NER component462attempts to identify grammars and lexical information that may be used to construe meaning with respect to text represented in text data input therein. The NER component462identifies portions of text represented in text data input into the NLU component260that correspond to a named entity that may be applicable to processing performed by an application290associated with the recognizer463implementing the NER component462. The NER component462(or other component of the NLU component260) may also determine whether a word refers to an entity that is not explicitly mentioned in the text, for example “him,” “her,” “it” or other anaphora, exophora or the like.

Each recognizer463, and more specifically each NER component462, may be associated with a particular grammar model and/or database476, a particular set of intents478, and a particular personalized lexicon4386. Each gazetteer484may include application-indexed lexical information associated with a particular user and/or device. For example, a Gazetteer A (484a) includes application-index lexical information486aato486an. A user's music application lexical information might include album titles, artist names, and song names, for example, whereas a user's contact list application lexical information might include the names of contacts. Since every user's music collection and contact list is presumably different, this personalized information improves entity resolution (performed by an entity resolution component570described below).

An NER component462applies grammar models476and lexical information486associated with the application290(associated with the recognizer463implementing the NER component462) to determine a mention of one or more entities in text data input therein. In this manner, the NER component462identifies “slots” (i.e., one or more particular words in text data) that may be needed for later processing. The NER component462may also label each slot with a type (e.g., noun, place, city, artist name, song name, etc.).

Each grammar model476includes the names of entities (i.e., nouns) commonly found in speech about the particular application290(i.e., generic terms) to which the grammar model476relates, whereas the lexical information486is personalized to the user and/or the device from which the input audio data211originated. For example, a grammar model476associated with a shopping application may include a database of words commonly used when people discuss shopping.

A downstream process called entity resolution (performed by an entity resolution component570described below) actually links a portion of text data to an actual specific entity known to the system. To perform entity resolution, the entity resolution component570may utilize gazetteer information (484a-484n) stored in an entity library storage482. The gazetteer information484may be used to match text represented in text data output by the ASR component250with different entities, such as song titles, contact names, etc. Gazetteers (484a-484n) may be linked to users (e.g., a particular gazetteer484may be associated with a specific user's music collection), may be linked to certain applications (e.g., a shopping application, a music application, a video application, a communications application, etc.), or may be organized in a variety of other ways. Entity resolution may also be performed by a specific application290, for example by a communication application determining which specific entity in a contact list is associated with a name mentioned in a call command.

Each recognizer463may also include an intent classification (IC) component464. The IC component464parses text data input thereto to determine an intent(s) associated with the application290(associated with the recognizer463implementing the IC component464) that potentially corresponds to the utterance represented in the text data. An intent corresponds to an action to be performed that is responsive to the utterance represented by the text data. The IC component464may communicate with a database478of words linked to intents. For example, a music intent database may link words and phrases such as “quiet,” “volume off,” and “mute” to a <Mute> intent. The IC component464identifies potential intents by comparing words and phrases in input text data to the words and phrases in an intents database478associated with the application290that is associated with the recognizer463implementing the IC component464.

The intents identifiable by a specific IC component464are linked to application-specific (i.e., the application290associated with the recognizer463implementing the IC component464) grammar frameworks476with “slots” to be filled. Each slot of a grammar framework476corresponds to a portion of the text data that the system believes corresponds to an entity. For example, a grammar framework476corresponding to a <PlayMusic> intent may correspond to text data sentence structures such as “Play {Artist Name},” “Play {Album Name},” “Play {Song name},” “Play {Song name} by {Artist Name},” etc. However, to make entity resolution more flexible, grammar frameworks476may not be structured as sentences, but rather based on associating slots with grammatical tags.

For example, an NER component462may parse text data to identify words as subject, object, verb, preposition, etc. based on grammar rules and/or models prior to recognizing named entities in the text data. An IC component464(implemented by the same recognizer463as the NER component462) may use the identified verb to identify an intent. The NER component462may then determine a grammar model476associated with the identified intent. For example, a grammar model476for a <PlayMusic> intent may specify a list of slots applicable to play the identified “object” and any object modifier (e.g., a prepositional phrase), such as {Artist Name}, {Album Name}, {Song name}, etc. The NER component462may then search corresponding fields in a lexicon486associated with the application290associated with the recognizer463implementing the NER component462, attempting to match words and phrases in text data the NER component462previously tagged as a grammatical object or object modifier with those identified in the lexicon486.

An NER component462may perform semantic tagging, which is the labeling of a word or combination of words according to their type/semantic meaning. An NER component462may parse text data using heuristic grammar rules, or a model may be constructed using techniques such as hidden Markov models, maximum entropy models, log linear models, conditional random fields (CRF), and the like. For example, an NER component462implemented by a music application recognizer463may parse and tag text data including text corresponding to “play mother's little helper by the rolling stones” as {Verb}: “Play,” {Object}: “mother's little helper,” {Object Preposition}: “by,” and {Object Modifier}: “the rolling stones.” The NER component462identifies “Play” as a verb based on a word database associated with the music application, which an IC component464(also implemented by the music application recognizer463) may determine corresponds to a <PlayMusic> intent. At this stage, no determination has been made as to the meaning of “mother's little helper” and “the rolling stones,” but based on grammar rules and models, the NER component462has determined that the text of these phrases relates to the grammatical object (i.e., entity) of the hypothesis represented in the text data.

The frameworks linked to the intent are then used to determine what database fields should be searched to determine the meaning of these phrases, such as searching a user's gazetteer484for similarity with the framework slots. For example, a framework for a <PlayMusic> intent might indicate to attempt to resolve the identified object based on {Artist Name}, {Album Name}, and {Song Name}, and another framework for the same intent might indicate to attempt to resolve the object modifier based on {Artist Name}, and resolve the object based on {Album Name} and {Song Name} linked to the identified {Artist Name}. If the search of the gazetteer484does not resolve a slot/field using gazetteer information, the NER component462may search a database of generic words associated with the application290(in the knowledge base472). For example, if the text data includes text corresponding to “play songs by the rolling stones,” after failing to determine an album name or song name called “songs” by “the rolling stones,” the NER component462may search the application vocabulary for the word “songs.” In the alternative, generic words may be checked before the gazetteer information, or both may be tried, potentially producing two different results.

An NER component462may tag text of a hypothesis represented in text data to attribute meaning to the hypothesis. For example, an NER component462may tag “play mother's little helper by the rolling stones” as: {application} Music, {intent} Play Music, {artist name} rolling stones, {media type} SONG, and {song title} mother's little helper. For further example, the NER component462may tag “play songs by the rolling stones” as: {application} Music, {intent} Play Music, {artist name} rolling stones, and {media type} SONG.

The NLU component260may generate cross-application N-best list data540including the tagged text data output by each recognizer463(as illustrated ifFIG. 5). A recognizer463may output tagged text data generated by an NER component462and an IC component464operated by the recognizer463, as described herein above. Each entry of tagged text data represented in the cross-application N-best list data540may be associated with a respective score indicating a likelihood that the tagged text data corresponds to the application290associated with the recognizer463from which the tagged text data was output. For example, the cross-application N-best list data540may be represented as:

The NLU component260may send the cross-application N-best list data540to a pruning component550. The pruning component550may sort the tagged text data represented in the cross-application N-best list data540according to their respective scores. The pruning component550may then perform score thresholding with respect to the cross-application N-best list data540. For example, the pruning component550may select tagged text data represented in the cross-application N-best list data540associated with a score satisfying (e.g., meeting and/or exceeding) a threshold score. The pruning component550may also or alternatively perform number of tagged text data thresholding. For example, the pruning component550may select a maximum threshold number of top scoring tagged text data. The pruning component550may generate cross-application N-best list data560including the selected tagged text data. The purpose of the pruning component550is to create a reduced list of tagged text data so that downstream, more resource intensive, processes may only operate on the tagged text data that most likely correspond to the utterance input to the system.

The NLU component260may also include a light slot filler component552. The light slot filler component552can take text from slots represented in the tagged text data output by the pruning component550and alter it to make the text more easily processed by downstream components. The light slot filler component552may perform low latency operations that do not involve heavy operations such as reference to a knowledge base. The purpose of the light slot filler component552is to replace words with other words or values that may be more easily understood by downstream system components. For example, if a tagged text data includes the word “tomorrow,” the light slot filler component552may replace the word “tomorrow” with an actual date for purposes of downstream processing. Similarly, the light slot filler component552may replace the word “CD” with “album” or the words “compact disc.” The replaced words are then included in the cross-application N-best list data560.

The NLU component260sends the cross-application N-best list data560to an entity resolution component570. The entity resolution component570can apply rules or other instructions to standardize labels or tokens from previous stages into an intent/slot representation. The precise transformation may depend on the application290. For example, for a travel application, the entity resolution component570may transform text data corresponding to “Boston airport” to the standard BOS three-letter code referring to the airport. The entity resolution component570can refer to a knowledge base that is used to specifically identify the precise entity referred to in each slot of each tagged text data represented in the cross-application N-best list data560. Specific intent/slot combinations may also be tied to a particular source, which may then be used to resolve the text data. In the example “play songs by the stones,” the entity resolution component570may reference a personal music catalog, Amazon Music account, a user profile302(described herein), or the like. The entity resolution component570may output data including an altered N-best list that is based on the cross-application N-best list represented in the cross-application N-best list data560, and that includes more detailed information (e.g., entity IDs) about the specific entities mentioned in the slots and/or more detailed slot data that can eventually be used by an application290. The NLU component260may include multiple entity resolution components570and each entity resolution component570may be specific to one or more applications290.

The entity resolution component570may not be successful in resolving every entity and filling every slot represented in the cross-application N-best list data560. This may result in the entity resolution component570outputting incomplete results. The NLU component260may include a final ranker component590that assigns a particular confidence score to each portion of tagged text data input therein. The confidence score of tagged text data may represent a confidence of the system in the NLU processing performed with respect to the tagged text data. The confidence score of particular tagged text data may be affected by whether the tagged text data has unfilled slots. For example, if tagged text data associated with a first application includes slots that are all filled/resolved, that tagged text data may be assigned a higher confidence score than other tagged text data including at least some slots that are unfilled/unresolved by the entity resolution component570.

The final ranker component590may apply re-scoring, biasing, or other techniques to determine the top scoring tagged text data. To do so, the final ranker component590may consider not only the data output by the entity resolution component570, but may also consider other data591. The other data591may include a variety of information. For example, the other data591may include application rating data. For example, if one application has a high rating, the final ranker component590may increase the score of tagged text data output by a recognizer463associated with that application. The other data591may also include information about applications that have been enabled by the user that input the utterance to the system. For example, the final ranker component590may assign higher scores to tagged text data output by recognizers463associated with enabled applications than tagged text data output by recognizers463associated with non-enabled applications. The other data591may also include data indicating system usage history, such as if the user that input the utterance to the system regularly uses a particular application or does so at particular times of day. The other data591may additionally include information indicating date, time, location, weather, type of device110, user ID, context, as well as other information. For example, the final ranker component590may consider when any particular application is currently active (e.g., music being played, a game being played, etc.).

Following final ranking, the NLU component260may output NLU results data585. The NLU component260may send the NLU results data585to the orchestrator component230, which sends the NLU results data585to the one or more applications290represented in the NLU results data585. For example, the NLU results data585may include first NLU results data585aincluding tagged text data associated with a first application, second NLU results data585bincluding tagged text data associated with a second application, etc. The NLU results data585may include tagged text data corresponding to the top scoring tagged text data as determined by the final ranker component590.

FIG. 6illustrates how the system may resolve a recipient user7post-NLU. The NLU component260(or another component of the server(s)120such as the orchestrator component230) sends the NLU results data585to a communications application290a. The NLU results data585includes a communications intent (e.g., a <Call> intent, a <SendMessage> intent, etc.) along with text data tagged to indicate one or more entities applicable to processing performed by the communications application290a. For example, the text data may be tagged to indicate a recipient user7.

In some instances, the recipient user7indicated in the tagged text data input to the communications application290amay be ambiguous. For example, the tagged text data may indicate the recipient user7as being “Joe” and a contact list associated with the originating user5may include more than one Joe. For further example, the tagged text data may indicating the recipient user7as being “Mom” and the contact list associated with the originating user5may include an entry for a person having a name “Mom” and an entry for a restaurant having a name “Mom's Favorite Country Cooking Restaurant.” In other instances, the recipient device110bindicated in the tagged text data input to the communications application290amay be ambiguous. For example, the tagged text data may indicate an exact recipient user “Joe Smith,” but the contact list associated with the originating user5may indicate more than one device associated with Joe Smith that is capable of receiving a communication. In these situations, the communications application290amay be unable to resolve the exact recipient user7and/or recipient device110b, resulting in the communications application290abeing unable to cause the communications server(s)125to establish communication between the originating device110aand an appropriate recipient device110b.

To resolve an ambiguous recipient user7and/or recipient device110b, the communications application290amay receive context data602representing a current context experienced by the originating user5and/or the originating device110a. The context data602may be unrelated to speech processing performed by the speech processing component240. The context data602may include time data indicating a current time or a time when the system received the utterance from the originating user5. The communications application290amay receive current time data from an electronic clock component maintained and/or operated by the server(s)120.

The context data602may also include location data indicating a location of the originating device110a. The communications application290amay receive the location data from a profile associated with the originating device110aor a user profile202indicating the originating device110a.

The context data602may also include user identifier (ID) data corresponding to a unique ID associated with the originating user5. The communications application290amay receive the user ID data from the user recognition component295.

The context data602may also include presence data indicating a number of individuals that are detected by the originating device110a. The context data602may also include data indicating content that is being presented on a display of the originating device110a. Other types of context data602are also possible.

The context data602may also include data representing previous messaging and calling information as well as geolocation information. For example, a contact list in the user's phone may represent a contact as “wife.” The pattern data604may indicate that the user routinely sends “wife” a message when the user is leaving work and calls “wife” when the user is in its car. If a user thereafter messages “wife” from the user's work and says “call Mary” when the user is in its car, the system may resolve “Mary” to the “wife” contact, in the user's contact list, based on the context data602and the pattern data604.

In an example, a user may routinely call a “mom” contact in a contact list on Sunday at 6:00 pm. The system may be configured to determine the user is intending to call “mom” on Sunday at 6:00 pm even if the user's pronunciation of “mom” deviates (for example if the user has a mouth full of food when speaking the command).

The communications application290amay receive pattern data604specific to the originating user5. The pattern data604may represent one or more patterns determined by the communications pattern component265with respect to previous communications interactions in which the originating user5was involved. The communications application290amay receive the pattern data204in response to the communications application290asending (either directly or via the orchestrator component230) the user ID data associated with the originating user5to the communications pattern component265.

The communications application290adetermines (630) a portion of the pattern data604associated with the context data602. For example, the communications application290amay determine the context data602indicates it is Sunday at 5:58 pm and the originating user5spoke the utterance to an originating device110alocated in a kitchen of the originating user's house. The communications application290amay determine a portion of the pattern data604representing a pattern wherein the originating user5routinely calls Joe Smith on Sundays around 6:00 pm using the originating device110alocated in the kitchen of the originating user's house. The communications application290amay determine the aforementioned portion of pattern data604is associated with the aforementioned context data602based on their substantial similarities. One skilled in the art should appreciate that other context data602may be associated with other portions of the pattern data604based on their similarities.

The communications application290adetermines (632) a recipient user7associated with the portion of the pattern data604associated with the context data602. That is, the communications application290adetermines the recipient user7the originating user5routinely called, as represented in the portion of the pattern data604.

The communications application290adetermines (634) a recipient device110bassociated with at least one of the recipient user7or the portion of the pattern data604associated with the context data602. The communications application290amay determine the recipient device110busing a user profile302associated with the recipient user7.

The communications application290asends (636) an indication representing the originating device110aand the recipient device110bto the communications server(s)125. The communications application290aalso sends (638) an indication of a type of communication to the communications server(s)125. The type of communication may be determined based on the intent representing the utterance. For example, if the utterance is represented by a <Call> intent, the type of communication may correspond to a call. For further example, if the utterance is represented by a <SendMessage> intent, the type of communication may correspond to a message.

If the communication corresponds to a call, the communications server(s)125may establish a two-way communication channel between the originating device110aand the recipient device110b. If the communication corresponds to a message, the communication server(s)125may establish a one-way communication channel through which the message content data is sent to the recipient device110b.

Prior to establishing a two-way communication channel between the originating device110aand the recipient device110b, the communications server(s)125or the communications application290amay generate output text data indicating the recipient user7and/or a unique identifier (e.g., phone number) associated with the recipient device110b. The communications server(s)125of the communications application290amay send the output text data to the TTS component280, which generates output audio data corresponding to synthesized speech corresponding to the output text data. The system may then cause the originating device110ato output audio corresponding to the output audio data. If the user inputs a second utterance corresponding to “Alexa, cancel,” “cancel, “Alexa, stop,” “stop,” or the like, the system may cause the communications server(s)125to abstain from establishing the two-way communication channel between the originating device110aand the recipient device110b. Outputting such a prompt to the originating user5enables the originating user5to prevent a call from being established with a user that the originating user5did not intend on communicating with. For example, such a situation may occur when the originating user5intends on perform a non-routine communication with a different user at a time when the originating user5usually makes a call to the recipient user7.

Once the communications application290adetermines a portion of the pattern data604associated with the context data602, the communications application290amay generate output text data prompting the originating user5regarding whether the originating user5wants the system to set a reminder with respect to the routine communication. The communications application290asends the output text data to the TTS component280, which generates output audio data corresponding to synthesized speech corresponding to the output text data. The system then causes the originating device110ato output audio corresponding to the output audio data. For example, the originating device110amay output synthesized speech corresponding to “I noticed you call Joe smith every Sunday at about 6:00 pm, do you want me to set a reminder?” If the user inputs an indication (e.g., a second utterance, an activation of a virtual button presented on a graphical user interface (GUI), input text response using a virtual keyboard presented on a GUI) that a reminder should be set, the system causes an electronic reminder application to generate a reminder based on the portion of the pattern data602. For example, if the portion of the pattern data602indicates the originating user5calls the recipient user7on Sundays at about 6:00 pm, the electronic reminder application may generate a reminder for every Sunday at 6:00 pm.

The reminder generated by the electronic reminder application may be location specific. For example, an originating user5may routinely call Joe Smith at about 6:00 pm using an originating device110alocated in a kitchen of the originating user's house. The reminder may be configured to only be triggered if conditions precedent are satisfied. According to the above example, the reminder may only be output to the originating user5if it is about 6:00 pm and the server(s)120receives data (e.g., image data or audio data) indicating the originating user5is present in the kitchen of the originating user's house. Image data may indicate the originating user5is present if the server(s)120determines a representation of the originating user5in the image data. Audio data may indicate the originating user5is present if the server(s)120determines speech of the originating user5in the audio data.

It should thus be appreciated that the reminder may further be user ID specific since the system may not want to output a reminder to a user that is not the originating user5associated with the reminder. For example, the server(s)120may determine a representation of the originating user5in image data, as opposed to simply determining a representation of a user in the image data, to prevent the reminder from being output to the wrong user. For further example, the server(s)120may determine speech of the originating user5in audio data, as opposed to simply determining speech of a user in the audio data, to prevent the reminder from being output to the wrong user.

The communications application290amay proactively indicate to the originating user5that the recipient user7is unavailable. After the communications application290adetermines the recipient device110b, the communications application290amay send a signal to the recipient device110b(or other information related to the desired recipient) to provide the system with data that may be processed to determine if the recipient user7is “present.” If the system receives image data from the recipient device110b, the system may process the image data to determine a representation of a user. If the system is unable to determine a representation of a user in the image data, the system may send an indication representing the recipient user7is unavailable to the communications application290a. If the system receives audio data from the recipient device110b, the system may process the audio data to determine speech of a user. If the system is unable to determine speech of a user in the audio data, the system may send an indication representing the recipient user7is unavailable to the communications application290a. Based on receiving the indication, the communications application290amay generate output text data indicating the recipient user7is unavailable. The communications application290asends the output text data to the TTS component280, which generates output audio data corresponding to synthesized speech corresponding to the output text data. The system then causes the originating device110ato output audio corresponding to the output audio data. For example, the originating device110amay output synthesized speech corresponding to “Joe Smith does not appear to be available for your routine call, would you like to send him a message,” “Joe Smith does not appear available to take your call at their kitchen device. Do you want to call his cell,” etc.

As described, the system may determine whether the recipient user7is unavailable based on image data or audio data provided by the recipient device110b. The system may alternatively determine whether the recipient user7is unavailable based on data provided by an alarm system associated with the same building (e.g., the recipient user's house) as the recipient device110b. For example, the system may receive an indication that the alarm system was activated prior to the system receiving the originating utterance from the originating user5. Based thereon, the communications application290amay generate output text data indicating the recipient user7is unavailable. The communications application290asends the output text data to the TTS component280, which generates output audio data corresponding to synthesized speech corresponding to the output text data. The system then causes the originating device110ato output audio corresponding to the output audio data. For example, the originating device110amay output synthesized speech corresponding to “Joe Smith does not appear to be home, would you like to send him a message,” “Joe smith does not appear to be home. Do you want to call his cell,” etc.

The system may be proactive with respect to a recipient user7. For example, the system may determine pattern data604indicates the recipient user7usually receives a call at about 6:00 pm every Sunday. At about 6:25 pm on Sundays the system may output a reminder that the recipient user7will likely be receiving a call based on context data602including time data and date data. For further example, the system may determine pattern data604indicates the recipient user7usually receives a call from an originating user5when the originating user5is cooking dinner. If the system determines the originating user5asks the system to output a recipe in the evening based on context data602including display content data and time data, the system may output synthesized speech to the recipient user7corresponding to “grandma usually calls you when she is cooking dinner and she just asked for a recipe,” for example. Yet further, for example, the system may determine pattern data604indicates the recipient user7usually receives a call from an originating user5at about 6:00 pm when the originating user5is in their vehicle. If the system determines the originating user5is present in their vehicle at 5:58 pm based on context data602including time data and presence data, the system may output synthesized speech to the recipient user7corresponding to “your wife usually calls you around 6 pm when she is in her car, it is 5:58 pm and she just entered her car.”

FIG. 7illustrates how the server(s)120may proactively remind a recipient user7of a likely future communication. The server(s)120may maintain a store of pattern data604and may continually monitor context data602of various users of the system. At some point, the system may determine (702) a portion of pattern data604, associated with an originating user5, that corresponds to context data602associated with a current context experienced by the originating user5. The server(s)120determines (632) a recipient user7associated with the portion of the pattern data604and determines (634) a recipient device110bassociated with at least one of the recipient user7or the portion of the pattern data604.

The server(s)120also generates (704) output text data indicating the recipient user7is likely to receive a communication. The server(s)120may perform (706) TTS on the output text data to generate output audio data. The server(s)120then causes (708) the recipient device110bto present text corresponding to the output text data and/or audio corresponding to the output audio data.

The system may use pattern data604and context data602to adjust NLU final rankings. For example, pattern data604may indicate a recipient user7routinely receives a call using a first communications application at about 6:00 pm on Sundays. On one Sunday at 5:55 pm, a second user (e.g., a family member of the recipient user7) may input an utterance to the system. The ASR component250may process input audio data corresponding to the utterance to generate input text data. The NLU component260may process the input text data to generate cross-application N-best list data560including first tagged text data associated with the first communications application and second tagged text data associated with a second communications application. The final ranker component590may receive pattern data604and context data602(which may be represented as other data591inFIG. 5). The final ranker component590may determine the pattern data604and the context data602indicate the recipient user7is expected to receive a call in 5 minutes. Based on this, the final ranker component590may decrease an NLU processing score associated with the tagged text data associated with the first communications application and/or increase an NLU processing score associated with the tagged text data associated with the second communications application. This results in the system causing the requested call to be conducted by the second communications application, thereby ensuring the first communications application (which is used to conduct the recipient user's routine call) is free to conduct the routine call.

FIG. 8illustrates how the system may user pattern data604and context data602to adjust NLU final rankings. The server(s)120receives (802) input audio data corresponding to an utterance. The ASR component250performs (804) ASR on the input audio data to generate input text data. Recognizers463perform (806) named entity recognition and intent classification to determine a first intent representing the utterance and a second intent representing the utterance. The first intent and the second intent may both correspond to <Call> intents. The first intent may be associated with a first communications application and the second intent may be associated with a second communications application. The pruning component550generates (808) cross-application N-best list data560representing first tagged text data associated with the first intent, the first communications application, and a first communications application NLU processing score; and second tagged text data associated with the second intent, the second communications application, and a second communications application NLU processing score. The final ranker component590receives (810) pattern data604and context data602(represented as other data591inFIG. 5). The final ranker component590determines (812) the pattern data604and the context data602indicate a likely future communication to be performed by the first communications application. Based on this, the final ranker component590generates (814) an adjusted first communications application NLU processing score by decreasing the first communications application NLU processing score and/or an adjusted second communications application NLU processing score by increasing the second communications application NLU processing score.

The processing performed with respect toFIG. 8may be user ID specific. Prior to performing NLU processing, the system determines the user that spoken the utterance. If the system determines the user that spoke the utterance is not the recipient user7represented in the pattern data604, the system may engage in the processes illustrated in and described with respect toFIG. 8. If the system determines the user that spoke the utterance is the recipient user7, the system may instead generate output text data corresponding to “you may receive a call shortly, do you want to proceed with making the requested call,” for example. The system may perform TTS on the output text data to generate output audio data. The system may then present output text corresponding to the output text data and/or output audio corresponding to the output audio data to the recipient user7.

The system may also ensure a recipient device110bis available to conduct a communication without affecting NLU ranking. Pattern data604may indicate a recipient user7routinely receives a call at about 6:00 pm on Sundays. The recipient user7may routinely enable a privacy mode of the system at about 6:00 pm every day. On Sunday, when the recipient user7attempts to enable the privacy mode, the system may, prior to enabling the privacy mode, present the recipient user7with text and/or audio corresponding to “you usually receive a call at about 6:00 pm, do you want to proceed with enabling privacy mode,” for example. If the recipient user7thereafter provides the system with an indication that the recipient user7wants to enable the privacy mode, the system may then enable the privacy mode.

FIG. 9illustrates how the server(s)120may ensure a recipient device110bis available to conduct a communication. The server(s)120receives (902) an indication that a user is requesting the system perform an action. For example, a user may request the system enable a privacy mode, start playing a movie, start playing music, etc. The server(s)120determines (904) a user ID associated with the user and/or a device ID associated with the device that sent the indication to the server(s)120. The server(s)120determines (906) pattern data604associated with the user ID and/or the device ID. The server(s)120also determines (908) context data602. The server(s)120determines (910) a recipient device110bassociated with the userID and/or device ID is likely to receive a communication based on the pattern data604and the context data602. The server(s)120also determine (911) that performance of the action represented in the received indication will (or will likely) affect the recipient device110bbeing able to perform the likely future communication. The server(s)120then generates (912) output text data indicating the recipient device110bis likely to receive a communication and soliciting the recipient user7regarding whether the recipient user7wants to the server(s)120to nonetheless perform the requested action represented in the indication. For example, the output text data may correspond to “I understand you want to enable privacy mode. You are likely to receive a call within the next 5 minutes from Joe. Can you confirm I should enable privacy mode?” The server(s)120may also perform (914) TTS on the output text data to generate output audio data. The server(s)120then causes (916), prior to performing the requested action, the recipient device110bto present output text corresponding to the output text data and/or output audio corresponding to the output audio data.

FIGS. 10-12Bbelow illustrated components that can be used to coordinate communications using a voice controlled system such as that described herein.FIG. 10illustrates an example of signaling to initiate a communication session according to the present disclosure. In one example configuration, the server(s)120is configured to enable voice commands (e.g., perform ASR, NLU, etc. to identify a voice command included in audio data), whereas the communications server(s)125is configured to enable communication sessions (e.g., using session initiation protocol (SIP)). For example, the communications server(s)125may send SIP messages to endpoints (e.g., the recipient device110b, etc.) in order to establish a communication session for sending and receiving audio data and/or video data. The communication session may use network protocols such as real-time transport protocol (RTP), RTP Control Protocol (RTCP), Web Real-Time communication (WebRTC) and/or the like. For example, the communications server(s)125may send SIP messages to initiate a single RTP media stream between two endpoints (e.g., direct RTP media stream between the originating device110aand the recipient device110b) and/or to initiate and facilitate RTP media streams between the two endpoints (e.g., RTP media streams between the originating device110aand the communications server(s)125and between the communications server(s)125and the recipient device110b). During a communication session, the communications server(s)125may initiate two media streams, with a first media stream corresponding to incoming audio data from the originating device110ato the recipient device110band a second media stream corresponding to outgoing audio data from the recipient device110bto the originating device110a, although for ease of explanation this may be illustrated as a single RTP media stream.

As illustrated inFIG. 10, the originating device110amay send (1002) audio data to the server(s)120and the server(s)120may determine (1004) call information using the audio data and may send (1006) the call information to the communications server(s)125. The server(s)120may determine the call information by performing ASR, NLU, etc., as discussed herein above, and the call information may include a data source name (DSN), a number from which to call, a number to which to call, encodings and/or additional information. For example, the server(s)120may identify from which phone number the originating user5would like to initiate the telephone call, to which phone number the originating user5would like to initiate the telephone call, from which device110the originating user5would like to perform the telephone call, etc.

WhileFIG. 10illustrates the server(s)120sending the call information to the communications server(s)125in a single step (e.g.,1006), the disclosure is not limited thereto. Instead, the server(s)120may send the call information to the originating device110aand the originating device110amay send the call information to the communications server(s)125in order to initiate the telephone call without departing from the disclosure. Thus, the server(s)120may not communicate directly with the communications server(s)125in step1006, but may instead instruct the originating device110ato connect to the communications server(s)125in order to initiate the telephone call.

The communications server(s)125may include an outbound SIP translator1032, an inbound SIP translator1034and a call state database1040. The outbound SIP translator1032may include logic to convert commands received from the server(s)120into SIP requests/responses and may handle sending outgoing SIP requests and sending responses to incoming SIP requests. After receiving the call information, the outbound SIP translator1032may persist (1008) a SIP dialog using the call state database1040. For example, the DSN may include information such as the name, location, and driver associated with the call state database1040(and, in some examples, a user identifier (ID) and password of the originating user5) and the outbound SIP translator1032may send a SIP dialog to the call state database1040regarding the communication session. The call state database1040may persist the call state if provided a device ID and one of a call ID or a dialog ID. The outbound SIP translator1032may send (1010) a SIP Invite to a SIP Endpoint1050, which may be the recipient device110b, a remote device, a Session Border Controller (SBC), or the like.

The inbound SIP translator1034may include logic to convert SIP requests/responses into commands to send to the server(s)120and may handle receiving incoming SIP requests and incoming SIP responses. The SIP endpoint1050may send (1012) a 100 TRYING message to the inbound SIP translator1034and may send (1014) a 180 RINGING message to the inbound SIP translator1034. The inbound SIP translator1034may update (1016) the SIP dialog using the call state database1040and may send (1018) a RINGING message to the server(s)120, which may send (1020) the RINGING message to the originating device110a.

When the communication session is accepted by the SIP endpoint1050, the SIP endpoint1050may send (1022) a 200 OK message to the inbound SIP translator1034, the inbound SIP translator1045may send (1024) a startSending message to the server(s)120, and the server(s)120may send (1026) the startSending message to the originating device110a. The startSending message may include information associated with an internet protocol (IP) address, a port, encoding, or the like required to initiate the communication session. Using the startSending message, the originating device110amay establish (1028) an RTP communication session with the SIP endpoint1050via the communications server(s)125.

WhileFIG. 10illustrates the communications server(s)125sending the RINGING message and the StartSending message to the originating device110avia the server(s)120, the disclosure is not limited thereto. Instead, steps1018and1020may be combined into a single step and the communications server(s)125may send the RINGING message directly to the originating device110awithout departing from the disclosure. Similarly, steps1024and1026may be combined into a single step and the communications server(s)125may send the StartSending message directly to the originating device110awithout departing from the disclosure. Thus, the communications server(s)125may communicate with the originating device110adirectly without using the server(s)120as an intermediary.

For ease of explanation, the disclosure illustrates the system using SIP. However, the disclosure is not limited thereto and the system may use any communication protocol for signaling and/or controlling communication sessions without departing from the disclosure. Similarly, while some descriptions of the communication sessions refer only to audio data, the disclosure is not limited thereto and the communication sessions may include audio data, video data and/or any other multimedia data without departing from the disclosure.

FIGS. 11A-111Billustrate examples of signaling to end a communication session according to the present disclosure. After establishing the RTP communication session1028between the originating device110aand the SIP endpoint1050, the RTP communication session may be ended by the originating user5inputting a command to end the telephone call to the originating device110a, as illustrated inFIG. 11A, or a the recipient user7inputting a command to end the telephone call to the SIP endpoint1050, as illustrated inFIG. 11B.

As illustrated inFIG. 11A, the originating device110amay send (1102) a state change message to the server(s)120and the server(s)120may send (1104) an end message to the communications server(s)125. The outbound SIP translator1032may update (1106) the session using the call state database1140and may send (1108) a SIP BYE message to the SIP endpoint1050. The SIP endpoint1050may send (1010) a 200 OK message to the inbound SIP translator1034and the inbound SIP translator1034may update (1112) the session using the call state database1140. In some examples, the inbound SIP translator1034may send the 200 OK message to the originating device110ato confirm that the communication session has been ended. Thus, the RTP communication session1028may be ended between the originating device110aand the SIP endpoint1050.

As illustrated inFIG. 11B, the SIP endpoint1050may send (1152) a SIP BYE message to the inbound SIP translator1034and the inbound SIP translator1034may update (1154) the session using the call state database1040. The inbound SIP translator1034may send (1156) a stopSending message to the server(s)120and the server(s)120may send (1158) the stopSending message to the originating device110a. The originating device110amay send (1160) a state change message to the server(s)120and the server(s)120may send (1162) an End message to the outbound SIP translator1032, the End message including a DSN. The outbound SIP translator1032may then update (1164) the session using the call state database1040, and send (1166) a 200 OK message to the SIP endpoint1050. Thus, the RTP communication session1028may be ended between the originating device110aand the SIP endpoint1050.

WhileFIGS. 11A and 11Billustrate the server(s)120acting as an intermediary between the originating device110aand the communications server(s)125, the disclosure is not limited thereto. Instead, steps1102and1104may be combined into a single step and the originating device110amay directly send the state change message and/or the End message to the communications server(s)125without departing from the disclosure. Similarly, steps1156and1158may be combined into a single step and the communications server(s)125may send the StopSending message directly to the originating device110awithout departing from the disclosure, and/or steps1160and1162may be combined into a single step and the originating device110amay directly send the state change message and/or the End message to the communications server(s)125without departing from the disclosure.

WhileFIGS. 10, 11A, and 11Billustrate the RTP communication session1028being established between the originating device110aand the SIP endpoint1050, the disclosure is not limited thereto and the RTP communication session1028may be established between the originating device110aand a telephone network associated with the SIP endpoint550without departing from the disclosure.

FIG. 12A-12Billustrate examples of establishing media streams between devices according to the present disclosure. In some examples, the originating device110amay have a publicly accessible IP address and may be configured to establish the RTP communication session directly with the SIP endpoint1050. To enable the originating device110ato establish the RTP communication session, the communications server(s)125may include Session Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs) server(s) (e.g., STUN server(s)1210). The STUN server(s)1210may be configured to allow NAT clients (e.g., device110behind a firewall) to setup telephone calls to a VoIP provider hosted outside of the local network by providing a public IP address, the type of NAT they are behind, and a port identifier associated by the NAT with a particular local port. As illustrated inFIG. 11A, the originating device110amay perform (1212) IP discovery using the STUN server(s)1210and may use this information to set up an RTP communication session1214(e.g., UDP communication) between the originating device110aand the SIP endpoint1050to establish a telephone call.

In some examples, the originating device110amay not have a publicly accessible IP address. For example, in some types of NAT the originating device110acannot route outside of the local network. To enable the originating device110ato establish an RTP communication session, the communications server(s)125may include Traversal Using relays around NAT (TURN) server(s)1220. The TURN server(s)1220may be configured to connect the originating device110ato the SIP endpoint1050when the originating device110ais behind a NAT. As illustrated inFIG. 12B, the originating device110amay establish (1222) an RTP session with the TURN server(s)1220and the TURN server(s)1220may establish an RTP session with the SIP endpoint1050. Thus, the originating device110amay communicate with the SIP endpoint1050via the TURN server(s)1220. For example, the originating device110amay send outgoing audio data to the communications server(s)125and the communications server(s)125may send the outgoing audio data to the SIP endpoint1050. Similarly, the SIP endpoint1050may send incoming audio data to the communications server(s)125and the communications server(s)125may send the incoming audio data to the originating device110a.

In some examples, the system may establish communication sessions using a combination of the STUN server(s)1210and the TURN server(s)1220. For example, a communication session may be more easily established/configured using the TURN server(s)1220, but may benefit from latency improvements using the STUN server(s)1210. Thus, the system may use the STUN server(s)1210when the communication session may be routed directly between two devices and may use the TURN server(s)1220for all other communication sessions. Additionally or alternatively, the system may use the STUN server(s)1210and/or the TURN server(s)1220selectively based on the communication session being established. For example, the system may use the STUN server(s)1210when establishing a communication session between two devices (e.g., point to point) within a single network (e.g., corporate LAN and/or WLAN), but may use the TURN server(s)1220when establishing a communication session between two devices on separate networks and/or three or more devices regardless of network(s). When the communication session goes from only two devices to three or more devices, the system may need to transition from the STUN server(s)1210to the TURN server(s)1220. Thus, the system may anticipate three or more devices being included in the communication session and may establish the communication session using the TURN server(s)1220.

FIG. 13is a block diagram conceptually illustrating a user device110that may be used with the system.FIG. 14is a block diagram conceptually illustrating example components of a remote device, such as the server(s)120, which may assist with ASR processing, NLU processing, TTS processing, etc. Multiple servers120may be included in the system, such as one or more servers120for performing ASR processing, one or more servers120for performing NLU processing, one or more server(s)120performing TTS processing, etc. In operation, each of these devices (or groups of devices) may include computer-readable and computer-executable instructions that reside on the respective device (110/120), as will be discussed further below.

Computer instructions for operating each device (110/120) and its various components may be executed by the respective device's controller(s)/processor(s) (1304/1404), using the memory (1306/1406) as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory (1306/1406), storage (1308/1408), or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.

Each device (110/120) includes input/output device interfaces (1302/1402). A variety of components may be connected through the input/output device interfaces (1302/1402), as will be discussed further below. Additionally, each device (110/120) may include an address/data bus (1324/1424) for conveying data among components of the respective device. Each component within a device (110/120) may also be directly connected to other components in addition to (or instead of) being connected to other components across the bus (1324/1424).

Referring toFIG. 10, the device110may include input/output device interfaces1302that connect to a variety of components such as an audio output component such as a speaker1312, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device110may also include an audio capture component. The audio capture component may be, for example, a microphone1320or array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. The device110may additionally include a display1316for displaying content.

The components of the device(s)110and the server(s)120may include their own dedicated processors, memory, and/or storage. Alternatively, one or more of the components of the device(s)110and the server(s)120may utilize the I/O interfaces (1302/1402), processor(s) (1304/1404), memory (1306/1406), and/or storage (1308/1408) of the device(s)110and server(s)120, respectively. Thus, the ASR component250may have its own I/O interface(s), processor(s), memory, and/or storage; the NLU component260may have its own I/O interface(s), processor(s), memory, and/or storage; and so forth for the various components discussed herein.

As illustrated inFIG. 15, multiple devices (110a-110h,120,125) may contain components of the system and the devices may be connected over a network(s)199. The network(s)199may include a local or private network or may include a wide network such as the Internet. Devices may be connected to the network(s)199through either wired or wireless connections. For example, a speech-detection device110a/b, a smart phone110c, a smart watch110d, a tablet computer110e, a vehicle110f, a display device110g, and/or a smart TV110hmay be connected to the network(s)199through a wireless service provider, over a WiFi or cellular network connection, or the like. Other devices are included as network-connected support devices, such as the server(s)120, communications server(s)125, or others. The support devices may connect to the network(s)199through a wired connection or wireless connection. Networked devices may capture audio using one-or-more built-in or connected microphones or other audio capture devices, with processing performed by ASR components, NLU components, or other components of the same device or another device connected via the network(s)199, such as the ASR component250, the NLU component260, etc. of one or more servers120.