Speech processing and multi-modal widgets

Techniques for performing speech processing using multi-modal widget information are described. A system may receive input data corresponding to a user input. The system may also receive widget context data corresponding to one or more multi-modal widgets active at a device. The system may use the widget context data to perform natural language understanding (NLU) processing with respect to the user input, and for selecting a skill component for responding to the user input. The system may send a widget identifier to the skill component when invoking the skill to respond to the user input.

BACKGROUND

Natural language processing systems have progressed to the point where humans can interact with and control computing devices using their voices. Such systems employ techniques to identify the words spoken by a user based on the various qualities of received input data. Speech recognition combined with natural language understanding processing techniques enable speech-based user control of computing devices to perform tasks based on the spoken inputs. Speech recognition and natural language understanding processing techniques are sometimes referred to collectively or separately as spoken language understanding (SLU) processing. SLU processing may be used by computers, hand-held devices, telephone computer systems, kiosks, and a wide variety of other devices to improve human-computer interactions.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics concerned with transforming audio data associated with speech into text representative of that speech. Similarly, natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics concerned with enabling computers to derive meaning from text input containing natural language. ASR and NLU are often used together as part of a speech processing system. Text-to-speech (TTS) is a field of concerning transforming textual data into audio data that is synthesized to resemble human speech.

Certain devices that have a display screen may present content to a user. Some such devices may be configured to present content from multiple applications via, what is referred to as, multi-modal widgets herein. A multi-modal widget may correspond to a portion of a display screen that presents visual content, that may automatically updates when appropriate, and that the user may interact with using touch inputs and/or voice inputs.

Multi-modal widgets, as used herein, may be glanceable, self-updating, and interactive view of content and functionality displayed on a device screen. Multi-modal widgets may offer ambient experiences (where various technology, systems, devices—and data gathered by them seamlessly interact and adapt to a user's needs), enable better multitasking on the system, and increase the discoverability of skills. For example, a developer can use the multi-modal widgets to surface latest content or upcoming events, interact with users without starting a full skill experience, and give users a quick way to go directly to the desired or most interesting part of a skill. In addition to viewing content, a user may also perform actions using a multi-modal widget. For example, a user can check an item off a list or add an item to a list using touch or voice inputs. Depending on the user action, the multi-modal widget may update inline (i.e. update the present content at the multi-modal widget) or start a full-screen experience (i e. open a new user interface screen).

A developer may create a multi-modal widget by defining a rendered document (via software code) and how information is to be presented visually via the rendered document, how instructions are to be executed with respect to the rendered document, etc. The developer may also define data sources from which the multi-modal widget can be populated. For example, a weather multi-modal widget may receive weather data from a weather service. A single multi-modal widget may receive data from multiple sources. The multi-modal widget may update the data by requesting updated data from the data source or based on the data source pushing updated data to the multi-modal widget. The rendered document definition may include instructions on how the data is to be updated.

A system may control a multi-modal widget at a device using a particular type of software language. In some embodiments, the system may control multi-modal widgets using a JSON-based HTML5 language. An example of such a software language is the Alexa Presentation Language (APL). Other types of software languages may be used to control multi-modal widgets. Using APL, a developer may create visual experiences to accompany a skill, such as, animations, graphics, images, slideshows, video, etc., which may be presented to a user via a multi-modal widget. A developer may create JSON files including software instructions to control the multi-modal widget. Such JSON files may be referred to herein as an APL document. The APL document may be invoked and downloaded to a user device. The user device may import images and other data indicated in the APL document and render the programmed experience at the user device.

The system may provide multi-modal widget templates that can be customized by a developer using the APL document. The multi-modal widget templates may correspond to different widget sizes that can be used to control the size of the multi-modal widget on the device screen. The multi-modal widgets templates may include a list template that can be used to display a list of items (e.g., a shopping list, a task list, etc.), which may include text and/or images. The multi-modal widgets templates may include a text wrapping template that can be used for text-based experiences (e.g., displaying tips, facts, instructions, etc.). Another multi-modal template may be an action button template that can be used to present content along with a button that a user can select to perform an indicated action. Another multi-modal template may be an image-caption template that can be used to present an image along with text. Yet another multi-modal template may be a photo template that can be used to present an image as focused-content.

Some devices may enable a user to interact with skills. As used herein, a “skill” may refer to software, that may be installed on a machine or a virtual machine (e.g., software that may be launched in a virtual instance when called or otherwise activated), configured to process natural language understanding (NLU) output data (e.g., including an intent and optionally one or more entities) and perform one or more actions in response thereto. What is referred to herein as a skill may sometimes be referred to as or otherwise incorporated into an application, bot, action, or the like. A group of skills of related functionality may be associated with a domain. For example, a first music skill and a second music skill may be associated with a music domain. A user may interact with skills using voice inputs and/or touch inputs. Skills may provide multi-modal widgets to enable the user to interact with the skill using touch inputs.

The present disclosure relates to performing speech processing in view of multi-modal widgets presented by a screen of a device. Techniques of the present disclosure enable a user to multitask and seamlessly switch between interactions with multiple multi-modal widgets. Techniques of the present disclosure also enables a user to switch between interacting with a skill corresponding to a multi-modal widget and another skill that may not have a corresponding multi-modal widget presented at the device.

A device or a system, of the present disclosure, can use contextual information of the multi-modal widgets being presented at the device to interpret natural language inputs provided by the user. The widget context data may indicate the content presently being displayed at the multi-modal widget. The widget context data may also include a name of the multi-modal widget, a skill associated with the multi-modal widget, a widget identifier and a widget instance identifier. The device/system may use the widget context data to determine an intent of a natural language input and entity data corresponding to the natural language input. For example, the device/system may determine, using the widget context data, that the intent of the natural language input corresponds to the multi-modal widget or that the entity data corresponds to the content being presently displayed at the multi-modal widget. The device/system may further use the widget context data to select a skill for processing with respect to the natural language input.

Multiple instances, presenting different content, of the same multi-modal widget may be presented at a device. For example, a device may display weather information for a first city via a first instance of a weather multi-modal widget, and the device may also display weather information for a second city via a second instance of the weather multi-modal widget. In some embodiments, the device/system determines which multi-modal widget instance, at the device, the natural language input corresponds to, and send an indication of the multi-modal widget instance to the skill. The skill may use the multi-modal widget instance to determine an output responsive to the natural language input. The skill may alternatively or additionally send a command to the device to present an output responsive to the natural language input via the multi-modal widget instance. For example, the device may present a first shopping list via a first instance of a list multi-modal widget and a second shopping list via a second instance of the list multi-modal widget. The device/system may determine that a user's voice input to: “remove [item] from my list” corresponds to the first shopping list, and may send an identifier for the first instance of the list multi-modal widget to a list skill, so that the list skill may perform an action, responsive to the user input, with respect to the first list. As part of the action, the list skill may cause the device to update the first instance of the list multi-modal widget (corresponding to the first shopping list) based on the received identifier for the first instance of the list multi-modal widget.

Using the widget context data, skills may determine how an output is to be presented to the user. For example, a skill may select from presenting an output within the widget's existing layout/display configuration (referred to herein as “inline”) or launching a full screen interface to present the output (i.e. opening a new screen/window to present the output). The skill may determine how to present the output based on the widget context data indicating the content being presently displayed at the device at the multi-modal widget. For example, if the output relates to the presently displayed content, then the skill may present the output inline. As another example, if the output does not relate to the presently displayed content, then the skill may present the output via a full-screen interface.

Teachings of the present disclosure may be configured to incorporate user permissions and may only be performed if approved by a user. As such, the systems, devices, components, and techniques described herein would be typically configured to restrict processing where appropriate and only process user data in a manner that ensures compliance with all appropriate laws, regulations, standards, and the like. The teachings of the present disclosure can be implemented on a geographic basis to ensure compliance with laws in various jurisdictions and entities in which the computing components and/or user are located.

FIG.1shows a system100configured to perform speech processing in view of multi-modal widgets presented at a device. As shown inFIG.1, the system100may include a device110, local to a user105, and a system120connected across one or more networks199. The network(s)199may include the Internet and/or any other wide- or local-area network, and may include wired, wireless, and/or cellular network hardware. Although the figures and discussion of the present disclosure illustrate certain steps in a particular order, the steps described may be performed in a different order (as well as certain steps removed or added) without departing from the present disclosure.

The system120may be a speech processing system configured to process spoken natural language inputs using ASR and NLU processing. The system120may include multiple components to facilitate speech processing, such as, an orchestrator component130, an ASR component150, a NLU component160, a skill selection component165, a skill request router188, and one or more skill components190. The system120may also include a profile storage170, a TTS component180, and a user recognition component195to facilitate processing of users inputs and generating outputs. One or more of the skill components190may be in (wired or wireless) communication with a skill system(s)125located remote/external to the system(s)120.

The device110may display content via one or more multi-modal widgets112. The content may include text, icons, images, video, selectable user interface elements (e.g., buttons, links, etc.), and the like. Each widget112may be associated with a same or different skill component190. Content at the widget112may be determined (including updated, refreshed, etc.) by the corresponding skill component190.

The user105may select which multi-modal widgets are to display content at the device110. The user105may select certain multi-modal widgets112to be active and presenting content continuously (e.g., on a daily basis). For example, the user105may select a weather multi-modal widget to be active daily, and the weather multi-modal widget may update weather content presented via the weather multi-modal widget. Such multi-modal widgets112may automatically activate after a reboot of the device110.

Referring toFIG.1, the user105may speak an input, and the device110may capture audio107representing the spoken input. For example, the user105may say “Alexa, add [item] to my shopping list” or “Alexa, show me weather for [city].” In other examples, the user105may provide another type of input (e.g., selection of a button, selection of displayed graphical interface elements, may perform a gesture, etc.). The device110may send (step 1) audio data (or other type of input data, such as, image data, text data, etc.) corresponding to the user input to the system120for processing. In particular, the orchestrator component130may receive the input data from the device110.

In response to receiving the user input from the user105, the device110may also send (step 2) widget context data114to the system120. The widget context data114may correspond to the multi-modal widgets112active at the device110(i.e., presenting content via the device110when the device110receives the user input). The widget context data114may include widget identifiers for the multi-modal widgets112. In some embodiments, the widget context data114may also include a skill identifier associated with the skill component190corresponding to the multi-modal widget112. For example, the widget context data114may include a first widget identifier for the multi-modal widget112a, a first skill identifier corresponding to the multi-modal widget112a, a second widget identifier for the multi-modal widget112b, and a second skill identifier corresponding to the multi-modal widget112b.FIG.2illustrates example data included in the widget context data114corresponding to the multi-modal widget112. In some embodiments, the device110may send first widget context data114acorresponding to the first multi-modal widget112a, and may send separate second widget context data114bcorresponding to the second multi-modal widget112b.

In some embodiments, the widget context data114may also include a time when the user105most recently/last interacted with the multi-modal widget112(as illustrated inFIG.2). This time may be represented, in some embodiments, as time elapsed since the last interaction. The device110may store such information based on receiving user inputs (e.g., touch inputs, voice inputs, etc.) from the user105corresponding to the multi-modal widget112.

In some embodiments, the widget context data114may also include a time when the multi-modal widget112was last updated (as illustrated inFIG.2). This time may be based on the last time when content was updated at the multi-modal widget112or other data corresponding to the multi-modal widget112was updated (e.g., update multi-modal widget software, update multi-modal widget interface, etc.). For example, this time may be based on a last time when the device110received a command, from a skill component190, to update the multi-modal widget112.

In some embodiments, the widget context data114may also include information on where the multi-modal widget112is positioned (referred to as position information herein) on the display screen of the device110(as illustrated inFIG.2). The position information may be indicated in terms of quadrants or portions. For example, the position information for the multi-modal widget112a(inFIG.1) may be “left side panel”, and the position information for the multi-modal widget112b(inFIG.1) may be “center” or “bottom half.” Other position information may be right side panel, top panel, bottom panel, center, bottom half, top half, etc. In other embodiments, the position information may be indicated as coordinates along x-axis and y-axis (e.g., x-y coordinates) or pixel coordinates corresponding to the display interface of the device110. The coordinates may be a set of four coordinates, each one corresponding to a corner of the multi-modal widget.

The content displayed via the multi-modal widget112may be scrollable (i.e. the user105may be able to scroll up, scroll down, scroll left, or scroll right) to view different portions of the content. In some embodiments, the widget context data114may also include scroll information (as illustrated inFIG.2).

The widget context data114, in some embodiments, may also include content (as illustrated inFIG.2) being displayed via the multi-modal widget112at the device110at the time the user input from the user105is received. Some multi-modal widgets112may not display an entirety of the content at the device110due to multi-modal widget size, screen size, etc. As such, the multi-modal widget112may display a portion of the content based on the multi-modal widget size and the screen size. For example, some multi-modal widgets may have multiple pages or tabs that the user105can select to view other portions of the content. As a further example, some multi-modal widgets may have a scroll bar so the user105can view other portions of the content. The widget context data114may include (or otherwise indicate) the portion of the content being displayed at the device110, as it is viewable by the user105.

The widget context data114may, in some embodiments, also include a layout type of the multi-modal widget112at the time the user input is received (as illustrated inFIG.2). The layout type may be full screen (when the multi-modal widget is in a full-screen mode), overlay (when the multi-modal widget is overlaid over other multi-modal widgets), non-full screen, etc.

The widget context data114, in some embodiments, may also include a focus type for the multi-modal widget112(as illustrated inFIG.2). The focus type may be touch focus (when the user105touches the multi-modal widget112before or while providing the user input), visual focus (when the multi-modal widget112is visible on the display screen of the device110while the user105provides the user input), or non-visual focus (when the multi-modal widget112is active but not visible on the display screen of the device110, and may be displayed on a different tab or page of the display screen of the device110, or may be at least partially obscured by an overlaid multi-modal widget).

The widget context data114, in some embodiments, may also include a user-defined name for the multi-modal widget112(as illustrated inFIG.2). For example, the user105may name a shopping list multi-modal widget as [list 1]. The user105may provide a spoken input or a touch input to name the multi-modal widget112.

In some embodiments, the information described above as being included in the widget context data114may be provided by the device110. In other embodiments, some or all of the above described information may be determined by the system120. For example, the device110may send the widget identifiers for the multi-modal widgets112presented at the device110, and the system120may determine the content being presented via the multi-modal widgets112. The system120may send a request to the skill component190corresponding to the multi-modal widget112to determine the content being presented via the multi-modal widget112associated with the widget identifier.

The orchestrator component130may store (step 3) the widget context data114in a widget session storage142. The widget context data114, in the widget session storage142, may be associated with a device identifier for the device110and/or a user identifier for the user105. The user identifier for the user105may be determined by the user recognition component195as described in detail below. The device identifier may be sent by the device110along with the widget context data114, or with the input data corresponding to the user input from the user105.

The widget session storage142may store historic widget context data corresponding to the user identifier associated with the user105, and for user identifiers associated with other users. The historic widget context data may correspond to one or more multi-modal widgets that may have been previously active at the device110, and may not be active currently. The historic widget context data may correspond to one or more multi-modal widgets that may have been enabled for the device110, and may be currently disabled. In addition to the widget context data, the widget session storage142may also store interaction data corresponding to a multi-modal widget. Such interaction data may relate to user inputs received at the system120and corresponding to the multi-modal widget, output data generated by a skill component and corresponding to the multi-modal widget (i.e. output data presented via the multi-modal widget), NLU data corresponding to the user inputs, selected skill identifier, time the user input was received, etc. The widget session storage142may associate the widget context data and the corresponding interaction data with a widget session identifier in the storage142. One or more components of the system120may use the data stored at the widget session storage142to perform its processing. For example, the skill selection component165may use the data stored at the widget session storage142(and corresponding to historic interactions between the user105and multi-modal widgets) to select a skill component for the instant user input (received in step 1). As another example, the NLU component160may use the data stored at the widget session storage142to determine NLU data corresponding to the instant user input.

In the case that the input data (received in step 1) is audio data, the orchestrator component130may send (step 4) the audio data to the ASR component150, and the ASR component150may process the audio data to determine ASR data (e.g., token data, text data, one or more ASR hypotheses including token or text data and corresponding confidence scores, etc.) corresponding to the words spoken by the user105. Details on how the ASR component150may process the audio data are described below. The ASR component may send (step 5) the ASR data to the orchestrator component130.

In some embodiments, the orchestrator component130may also send (at step 4) the widget context data114to the ASR component150. The ASR component150may use the widget context data114to perform ASR processing. For example, the ASR component150may employ one technique involving boosting of one or more words (represented as text or token data) included in the widget context data114, such as, the name of the multi-modal widget112, words included in the content presented at the multi-modal widget112, etc. Using such as a technique may enable the ASR component150to recognize certain rare or personalized words that may be included in the spoken input from the user105. In some embodiments, the ASR component150may determine embedding data corresponding to the words included in the widget context data114, and use the embedding data to determine the ASR data corresponding to the audio data.

The orchestrator component130may send (step 6) the ASR data and the widget context data114to the NLU component160. The NLU component160may determine NLU data corresponding to the user input, where the NLU data may include an intent and one or more entities. The NLU component160may use the widget context data114to determine the intent and/or the one or more entities. In some cases, the NLU component160may determine the intent based on which multi-modal widgets112are presented at the device110. In some cases, the NLU component160may determine an entity based on the content being displayed at the multi-modal widget112. In some cases, the NLU component160may determine the NLU data (intent and/or entity) based on the focus information for the multi-modal widget112, content being displayed via the multi-modal widget112, time of last interaction with the multi-modal widget112, etc. For example, the device110may present a music multi-modal widget displaying information about a [song name], and the user105may say “Play song.” In this example, the NLU component160may determine, using the ASR data corresponding to the spoken input and the widget context data114, the following NLU data: {intent: <PlayMusic>; entity type: <SongName> entity: “[song name]”}. The entity [song name] may be based on the widget context data114indicating that the multi-modal widget112displaying information about the [song name]. In some embodiments, the widget context data114may be used to target certain Finite-state Transducers (FSTs) implemented by the NLU component160. For example, the NLU component160may implement FSTs specific to certain skills, and based on the widget context data114indicating which multi-modal widgets112, and in turn which skills, are active/enabled at the device110, the NLU component160may use or boost the corresponding FSTs. Further details about how the NLU component160determines NLU data are described below in relation toFIG.4. The NLU component160may send (step 7) the NLU data, corresponding to the user input, to the orchestrator component130.

The orchestrator component130may send (step 8) the NLU data and the widget context data114to the skill selection component165. The skill selection component165may be configured to determine which skill component190is capable of responding to the user input. The skill selection component165may make this determination based on which skill component190is capable of processing the intent and the entity data included in the NLU data. The skill selection component165may make this determination further based on the information included in the widget context data114. In selecting the skill component190, the skill selection component165may determine which particular multi-modal widget112the user input corresponds to. Based on determining which multi-modal widget112the user input corresponds to, the skill selection component165may select the skill component190to respond to the user input. In some embodiments, the skill selection component165may select the skill component190associated with the skill identifier included in the widget context data114as corresponding to the multi-modal widget112. For example, if the skill selection component165determines that the user input corresponds to the multi-modal widget112a, then the skill selection component165may select the skill component190aassociated with the first skill identifier corresponding to the multi-modal widget112a(as indicated in the widget context data114).

The skill selection component165may determine which multi-modal widget112corresponds to the user input based on the intent and/or the entity included in the NLU data. For example, if the intent is <PlayMusic>, then the skill selection component165may determine that the music multi-modal widget, presented at the device110, corresponds to the user input.

In some cases, the device110may present multiple instances of the same multi-modal widget. In some embodiments, the skill selection component165may determine which multi-modal widget instance corresponds to the user input. The skill selection component165may make this determination based on the content displayed via each multi-modal widget instance, the content that is visible on the device screen when the user105provided the user input, the time since last interaction, the focus information for the multi-modal widget, user-defined name, and/or other information included in the widget context data114. For example, the multi-modal widgets112aand112bmay be different instances of a shopping list multi-modal widget, and the user105may say “remove [item] from my list.” In this example, the skill selection component165may determine that the content displayed via the multi-modal widget112aincludes the [item], and may select the multi-modal widget112aas corresponding to the user input. In another example, a first list at a first instance of a shopping list multi-modal widget may be named by the user105[list 1], a second list at a second instance of the shopping list multi-modal widget may be named by the user105[list 2], and the user105may say “add [item] to my [list 1].” In this example, the skill selection component165may determine that the user input corresponds to the first shopping list multi-modal widget.

In some embodiments, the skill selection component165may output a widget identifier associated with the multi-modal widget112that is determined to correspond to the user input. In some embodiments, the skill selection component165may output more than one widget identifier if the skill selection component165determines the user input corresponds to more than one multi-modal widget112.

The skill selection component165may send (step 9) a skill identifier associated with the skill component190to the orchestrator component130. In some embodiments, the skill selection component165may also send a widget identifier associated with the multi-modal widget112determined to correspond to the user input.

The orchestrator component130may send (step 10) the skill identifier, the widget identifier and the NLU data to the skill request router component188. The orchestrator component130may also send a command to the skill request router component188to send the NLU data to the skill component190associated with the skill identifier, so that the skill component190may process with respect to the user input.

In some embodiments, the skill request router component188may determine whether data, to be sent to the skill component190and indicated by the orchestrator component130, is appropriate to send to the skill component190. As described herein, the system120may implement more than one skill component190, each of which may be configured to perform certain functionalities. When a particular skill component190ais to be invoked for a user input, the other skill components190are not to be invoked and are not to receive the NLU data or other data corresponding to the user input.

In some embodiments, the orchestrator component130may also send the widget context data114to the skill request router component188for sending to the skill component190. The skill request router component188may determine which portion of the widget context data114is to be sent to the skill component190. As described herein, the widget context data114may include data corresponding to all multi-modal widgets112active or enabled at the device110. The skill request router component188may determine widget context data116to be a portion of the widget context data114that corresponds to the widget identifier and/or the skill identifier received from the orchestrator component130at step 10. In this manner, the skill request router188may not send, to the skill component190, all the content being presented at the device110. This may prevent the skill component190from being able to identify content being presented at the device110and which the skill component190does not need to process (e.g., multi-modal widget content corresponding to a different skill component).

As used herein, a multi-modal widget may be “active” when the multi-modal widget is presenting content at the device110. In other words, when the user105adds the multi-modal widget to a GUI (e.g., home screen, another tab, etc.) of the device110. In some cases, the GUI of the device110may include multiple tabs or pages that the user105can scroll or switch to view other content. The user105may add one or more multi-modal widgets to a first tab/page, and add other multi-modal widgets to another tab/page.

As used herein, a multi-modal widget may be enabled/installed when the user105downloads a multi-modal widget to the device110. Such enabled/installed multi-modal widgets may not present content via the device110till the user105adds the multi-modal widget to the GUI of the device110.

In some embodiments, the functionalities described herein may take into consideration active multi-modal widgets to determine NLU data corresponding to a user input and select a skill component190to respond to the user input. In some embodiments, the functionalities described herein may also take into consideration enabled/installed multi-modal widgets. For example, the user105may say “add [multi-modal widget name] to the home screen,” in which case the NLU component160may determine entity data to be [multi-modal widget name] which may correspond to one of the enabled/installed multi-modal widgets, and the skill selection component165may select the skill component190corresponding to the particular enabled/installed multi-modal widget.

In some embodiments, the skill component190may store/track content data being presented via the multi-modal widget112at the device110. For example, the skill component190may store the content data when the content data is sent (via the orchestrator130) to the device110for output via the multi-modal widget112. In some cases, the multi-modal widget112may be scrollable or may update dynamically (e.g., update content based on time, etc.). In such cases, the content presently displayed at the device110and presently visible to the user105may be a portion of the content data that the skill component190is storing/tracking. For example, the skill component190may send weather data corresponding to multiple days for display via a weather multi-modal widget. The weather multi-modal widget may have a first view/tab that displays weather for the present day, a second view/tab that displays weather for the next day, and third view/tab that displays weather for multiple days. The user105may select, for example, the first view/tab, and thus weather data for the present day (which is a portion of the weather data sent by the skill component190) may be visible to the user105. In such cases, the device110is able to provide the presently visible content data, via the widget context data114, to the system120, which the skill component190(and other components of the system120) may use for processing.

The skill request router component188may send (step 11) the NLU data corresponding to the user input, the widget context data116, the widget identifier, and a command to process with respect to the NLU data to the skill component190. The skill component190may determine an output responsive to the user input based on the NLU data. In some embodiments, the skill component190may determine how the output is to be presented to the user105. Such determination may be based on the widget context data116and the widget identifier. For example, the skill component190may determine that the output is to be presented via the multi-modal widget112associated with the widget identifier. As another example, the skill component190may determine to launch another multi-modal widget at the device110to present the output. As yet another example, the skill component190may determine to present the output via a full-screen interface, where the full-screen interface may cause one or more of the multi-modal widgets112to appear in the background, or the full-screen interface may be presented on another tab/page of the display screen of the device110. As yet another example, the skill component190may determine to present the output via an overlay interface that may be presented on-top of the multi-modal widgets112and may cover one or more of the multi-modal widgets112or cover portions of one or more of the multi-modal widgets112.

The skill component190may also determine a type of output to be presented to the user105. For example, the skill component190may determine to present an audio-only output, a visual-only output (including text, image, video, icons, other graphical interface elements, etc.), or an audiovisual output. In some cases, the skill component190may determine to output a natural language output. In such cases, the skill component190may send text data or SSML tagged data, corresponding to the natural language output, to the orchestrator component130. The text data or SSML tagged data may be processed using the TTS component180to generate audio data representing synthesized speech, as described in detail below.

The skill component190may send (step 12) output data responsive to the user input to the orchestrator component130. The skill component190may also send (to the orchestrator component130) an indication of how the output data is to be presented to the user105and/or the type of output to be presented to the user105. The orchestrator component130may send (step 13) the output data to the device110, along with a directive/command that causes the device110to present the output in the manner specified by the skill component190. For example, if the indication from the skill component190indicates the output is to be presented via the multi-modal widget112a, then, in response to receiving the output data from the orchestrator130, the device110may update the content of the multi-modal widget112a. As another example, if the indication from the skill component190indicates the output is to be presented via a full-screen interface, then, in response to receiving the output data from the orchestrator130, the device110may launch a full-screen interface to present the output data. The skill component190, in some embodiments, may send (at step 12) an APL document including the output data, and the indication of how the output data is to be presented.

In this manner, the system120may determine NLU data and may select a skill component corresponding to a user input using widget context data corresponding to one or more multi-modal widgets being presented at the device110that captures the user input.

The widget context data114may be provided by the device110to the system120each time a user input is received and sent to the system120. In some cases, the widget context data114may remain the same between user inputs, if content at the multi-modal widgets112has not updated. In some cases, the widget context data114may remain the same after the device110is rebooted (restarted) if the content at the multi-modal widgets112has not updated.

Each skill component190may have multiple multi-modal widgets to provide different or related functionalities. The device110may present multiple instances of the same multi-modal widget. In some embodiments, each multi-modal widget112may be associated with a widget instance identifier, a widget identifier, and a skill identifier. In some embodiments, the skill selection component165may determine the widget instance identifier and the widget identifier corresponding to the user input, and may send the widget instance identifier and the widget identifier to the orchestrator component130.

In some embodiments, the widget context data114, sent by the device110to the system120, may include context data corresponding to multi-modal widgets that are not active at the device110but are installed or enabled, by the user105, at the device110.

In some embodiments, a component of the system120may request context data to perform its processing. Such context data may include a device identifier for the device110, user identifier for the user105, profile data for the user105from the profile storage170, time the user input is received, device location, etc. Such context data may also include widget context data114stored at the widget session storage142.

In some embodiments, the system120may initiate a skill session when a user input from the user105is received. The skill session may be used to track data determined during processing of the user input, context data corresponding to the user input, and other data. The skill session may be active while the system120is processing with respect to the user input. Once the system120presents an output responsive to the user input (e.g., at step 13) the skill session may be terminated.

In some embodiments, the system120may initiate a widget session when the multi-modal widget112is enabled at the device110. In other embodiments, the system120may initiate a widget session when the multi-modal widget112is active on the display screen of the device110. In some embodiments, the device110may send the widget context data114when the multi-modal widget112becomes enabled or active at the device110. The system120may store the widget context data114in the widget session storage142, associated with the device identifier for the device110and/or the user identifier for the user105. While the multi-modal widget112is installed/enabled or is active, the widget session may be active. When the multi-modal widget112is disabled or is no longer active at the display screen of the device110, the widget session may be terminated. The widget session may be used to track widget context data corresponding to the multi-modal widget112. The widget context data may be updated/refreshed when the multi-modal widget112is updated at the device110(e.g., with respect to content, display position, receipt of user input, etc.). Other components of the system120may request the widget context data to perform processing. In some cases, a lifecycle of a widget session may be longer than a skill session, as the skill session may terminate when an interaction ends (e.g., the system presents an output responsive to a user input), while the widget session may terminate when the multi-modal widget is disabled or no longer active at the device110.

In some embodiments, the data stored at the widget session storage142(and corresponding to historic interactions between the user105and the multi-modal widgets112) may be used by the NLU component160and the skill selection component165to determine how to respond to a user input corresponding to a multi-modal widget. For example, the user105may say “save that recipe” intending to save a recipe being displayed via the multi-modal widget112aat the device110. Based on processing the user input as described herein, the system120performs the action of saving the recipe displayed at the multi-modal widget112a. Subsequently (after some time after saying “save that recipe”) the user105may say “unsave that recipe” or “delete that recipe, I changed my mind” or other similar inputs. In this case, the NLU component160may determine a “reverse action” or “undo action” intent corresponding to the subsequent user input. Using the data stored at the widget session storage142, the NLU component160may determine that the intent corresponds to the multi-modal widget112abased on the historic interaction data indicating that a recipe was previously saved with respect to the multi-modal widget112a. The skill selection component165may select the skill component190, corresponding to the multi-modal widget112a, to respond to the subsequent user input (e.g., perform the action of deleting the recipe).

The system120may use other components illustrated inFIG.1. The various components shown inFIG.1may be located on a same or different physical devices. Communication between various components may occur directly or across the network(s)199.

A microphone or array of microphones (of or otherwise associated with the device110) may capture audio. The device110processes audio data, representing the audio, to determine whether speech is detected. The device110may use various techniques to determine whether audio data includes speech. In some examples, the device110may apply voice activity detection (VAD) techniques. Such techniques may determine whether speech is present in audio data based on various quantitative aspects of the audio data, such as the spectral slope between one or more frames of the audio data, the energy levels of the audio data in one or more spectral bands, the signal-to-noise ratios of the audio data in one or more spectral bands, or other quantitative aspects. In other examples, the device110may implement a classifier configured to distinguish speech from background noise. The classifier may be implemented by techniques such as linear classifiers, support vector machines, and decision trees. In still other examples, the device110may apply Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques to compare the audio data to one or more acoustic models in storage, which acoustic models may include models corresponding to speech, noise (e.g., environmental noise or background noise), or silence. Still other techniques may be used to determine whether speech is present in audio data.

Once speech is detected in audio data, the device110may determine if the speech is directed at the device110/system120. In at least some embodiments, such determination may be made using a wakeword detection component320(shown inFIG.3). The wakeword detection component320may be configured to detect various wakewords. In at least some examples, each wakeword may correspond to a name of a different digital assistant. An example wakeword/digital assistant name is “Alexa.”

Wakeword detection is typically performed without performing linguistic analysis, textual analysis, or semantic analysis. Instead, the audio data is analyzed to determine if specific characteristics of the audio data match preconfigured acoustic waveforms, audio signatures, or other data corresponding to a wakeword.

Once the wakeword detection component320detects a wakeword, the device110may “wake” and begin transmitting audio data311, representing the audio, to the system120. The audio data311may include the detected wakeword, or the device110may remove the portion of the audio data, corresponding to the detected wakeword, prior to sending the audio data311to the system120.

Referring toFIG.1, the orchestrator component130may be configured to, among other things, coordinate data transmissions between components of the system120. The orchestrator component130may receive audio data from the device110, and send the audio data to the ASR component150.

The ASR component150transcribes the audio data into ASR output data including one or more ASR hypotheses. An ASR hypothesis may be configured as a textual interpretation of the speech in the audio data, or may be configured in another manner, such as one or more tokens. Each ASR hypothesis may represent a different likely interpretation of the speech in the audio data. Each ASR hypothesis may be associated with a score (e.g., confidence score, probability score, or the like) representing the associated ASR hypothesis correctly represents the speech in the audio data.

The ASR component150interprets the speech in the audio data based on a similarity between the audio data and pre-established language models. For example, the ASR component150may compare the audio data311with models for sounds (e.g., subword units, such as phonemes, etc.) and sequences of sounds to identify words that match the sequence of sounds of the speech represented in the audio data.

In at least some instances, instead of the device110receiving a spoken natural language input, the device110may receive a textual (e.g., types) natural language input. The device110may determine text data representing the textual natural language input, and may send the text data to the system120, wherein the text data is received by the orchestrator component130. The orchestrator component130may send the text data or ASR output data, depending on the type of natural language input received, to the NLU component160.

The NLU component160processes the ASR output data or text data to determine one or more NLU hypotheses embodied in NLU output data. The NLU component160may perform intent classification (IC) processing on the ASR output data or text data to determine an intent of the natural language input. An intent corresponds to an action to be performed that is responsive to the natural language input. To perform IC processing, the NLU component160may communicate with a database of words linked to intents. For example, a music intent database may link words and phrases such as “quiet,” “volume off,” and “mute” to a <Mute> intent. The NLU component160identifies intents by comparing words and phrases in ASR output data or text data to the words and phrases in an intents database. In some embodiments, the NLU component160may communicate with multiple intents databases, with each intents database corresponding to one or more intents associated with a particular skill.

For example, IC processing of the natural language input “play my workout playlist” may determine an intent of <PlayMusic>. For further example, IC processing of the natural language input “call mom” may determine an intent of <Call>. In another example, IC processing of the natural language input “call mom using video” may determine an intent of <VideoCall>. In yet another example, IC processing of the natural language input “what is today's weather” may determine an intent of <OutputWeather>.

The NLU component160may also perform named entity recognition (NER) processing on the ASR output data or text data to determine one or more portions, sometimes referred to as slots, of the natural language input that may be needed for post-NLU processing (e.g., processing performed by a skill). For example, NER processing of the natural language input “play [song name]” may determine an entity type of “SongName” and an entity value corresponding to the indicated song name. For further example, NER processing of the natural language input “call mom” may determine an entity type of “Recipient” and an entity value corresponding to “mom.” In another example, NER processing of the natural language input “what is today's weather” may determine an entity type of “Date” and an entity value of “today.”

In at least some embodiments, the intents identifiable by the NLU component160may be linked to one or more grammar frameworks with entity types to be populated with entity values. Each entity type of a grammar framework corresponds to a portion of ASR output data or text data that the NLU component160believes corresponds to an entity value. For example, a grammar framework corresponding to a <PlayMusic> intent may correspond to sentence structures such as “Play {Artist Name},” “Play {Album Name},” “Play {Song name},” “Play {Song name} by {Artist Name},” etc.

For example, the NLU component160may perform NER processing to identify words in ASR output data or text data as subject, object, verb, preposition, etc. based on grammar rules and/or models. Then, the NLU component160may perform IC processing using the identified verb to identify an intent. Thereafter, the NLU component160may again perform NER processing to determine a grammar model associated with the identified intent. For example, a grammar model for a <PlayMusic> intent may specify a list of entity types applicable to play the identified “object” and any object modifier (e.g., a prepositional phrase), such as {Artist Name}, {Album Name}, {Song name}, etc. The NER processing may then involve searching corresponding fields in a lexicon, attempting to match words and phrases in the ASR output data that NER processing previously tagged as a grammatical object or object modifier with those identified in the lexicon.

NER processing may include semantic tagging, which is the labeling of a word or combination of words according to their type/semantic meaning. NER processing may include parsing ASR output data or text data using heuristic grammar rules, or a model may be constructed using techniques such as hidden Markov models, maximum entropy models, log linear models, conditional random fields (CRFs), and the like. For example, NER processing with respect to a music skill may include parsing and tagging ASR output data or text data corresponding to “play mother's little helper by the rolling stones” as {Verb}: “Play,” {Object}: “mother's little helper,” {Object Preposition}: “by,” and {Object Modifier}: “the rolling stones.” The NER processing may identify “Play” as a verb based on a word database associated with the music skill, which IC processing determines corresponds to a <PlayMusic> intent.

The NLU component160may generate NLU output data including one or more NLU hypotheses, with each NLU hypothesis including an intent and optionally one or more entity types and corresponding entity values. In some embodiments, the NLU component160may perform IC processing and NER processing with respect to different skills. One skill may support the same or different intents than another skill. Thus, the NLU output data may include multiple NLU hypotheses, with each NLU hypothesis corresponding to IC processing and NER processing performed on the ASR output or text data with respect to a different skill.

As described above, the system120may perform speech processing using two different components (e.g., the ASR component150and the NLU component160). In at least some embodiments, the system120may implement a spoken language understanding (SLU) component configured to process audio data311to determine NLU output data.

The SLU component may be equivalent to a combination of the ASR component150and the NLU component160. Yet, the SLU component may process audio data311and directly determine the NLU output data, without an intermediate step of generating ASR output data. As such, the SLU component may take audio data311representing a spoken natural language input and attempt to make a semantic interpretation of the spoken natural language input. That is, the SLU component may determine a meaning associated with the spoken natural language input and then implement that meaning. For example, the SLU component may interpret audio data311representing a spoken natural language input in order to derive a desired action. The SLU component may output a most likely NLU hypothesis, or multiple NLU hypotheses associated with respective confidence or other scores (such as probability scores, etc.).

The system120may include one or more skill components190and/or may communicate with one or more skills125. A “skill” may refer to software, that may be placed on a machine or a virtual machine (e.g., software that may be launched in a virtual instance when called), configured to process NLU output data and perform one or more actions in response thereto. For example, for NLU output data including a <PlayMusic> intent, an “artist” entity type, and an artist name as an entity value, a music skill may be called to output music sung by the indicated artist. For further example, for NLU output data including a <TurnOn> intent, a “device” entity type, and an entity value of “lights,” a smart home skill may be called to cause one or more “smart” lights to operate in an “on” state. In another example, for NLU output data including an <OutputWeather> intent, a “location” entity type, and an entity value corresponding to a geographic location of the device110, a weather skill may be called to output weather information for the geographic location. For further example, for NLU output data including a <BookRide> intent, a taxi skill may be called to book a requested ride. In another example, for NLU output data including a <BuyPizza> intent, a restaurant skill may be called to place an order for a pizza.

A skill component190may operate in conjunction between the system120and other devices, such as the device110, a restaurant electronic ordering system, a taxi electronic booking system, etc. in order to complete certain functions. Inputs to a skill component190may come from speech processing interactions or through other interactions or input sources.

A skill component190may be associated with a domain, a non-limiting list of which includes a smart home domain, a music domain, a video domain, a weather domain, a communications domain, a flash briefing domain, a shopping domain, and a custom domain.

The system120may include a TTS component180that generates audio data including synthesized speech. The data input to the TTS component180may come from a skill component190, the orchestrator component130, or another component of the system120. The data input to the TTS component180may be text data or synthesized speech markup language (SSML) tagged data.

In one method of synthesis called unit selection, the TTS component180matches input data against a database of recorded speech. The TTS component180selects matching units of recorded speech and concatenates the units together to form audio data. In another method of synthesis called parametric synthesis, the TTS component180varies parameters such as frequency, volume, and noise to determine audio data including an artificial speech waveform. Parametric synthesis uses a computerized voice generator, sometimes called a vocoder.

The system120may include a user recognition component195. The user recognition component195may recognize one or more users using various data. The user recognition component195may take as input the audio data311. The user recognition component195may perform user recognition by comparing speech characteristics, in the audio data311, to stored speech characteristics of users. The user recognition component195may additionally or alternatively perform user recognition by comparing biometric data (e.g., fingerprint data, iris data, retina data, etc.), received by the system120in correlation with a natural language input, to stored biometric data of users. The user recognition component195may additionally or alternatively perform user recognition by comparing image data (e.g., including a representation of at least a feature of a user), received by the system120in correlation with a natural language input, with stored image data including representations of features of different users. The user recognition component195may perform other or additional user recognition processes, including those known in the art. For a particular natural language input, the user recognition component195may perform processing with respect to stored data of users associated with the device110that received the natural language input.

The user recognition component195determines whether a natural language input originated from a particular user. For example, the user recognition component195may determine a first value representing a likelihood that a natural language input originated from a first user, a second value representing a likelihood that the natural language input originated from a second user, etc. The user recognition component195may also determine an overall confidence regarding the accuracy of user recognition processing.

The user recognition component195may output a single user identifier corresponding to the most likely user that originated the natural language input. Alternatively, the user recognition component195may output multiple user identifiers (e.g., in the form of an N-best list) with respective values representing likelihoods of respective users originating the natural language input. The output of the user recognition component195may be used to inform NLU processing, processing performed by a skill125, as well as processing performed by other components of the system120and/or other systems.

The system120may include profile storage170. The profile storage170may include a variety of data related to individual users, groups of users, devices, etc. that interact with the system120. As used herein, a “profile” refers to a set of data associated with a user, group of users, device, etc. The data of a profile may include preferences specific to the user, group of users, device, etc.; input and output capabilities of one or more devices; internet connectivity data; user bibliographic data; subscription data; skill enablement data; and/or other data.

The profile storage170may include one or more user profiles. Each user profile may be associated with a different user identifier. Each user profile may include various user identifying data (e.g., name, gender, address, language(s), etc.). Each user profile may also include preferences of the user. Each user profile may include one or more device identifiers, each representing a respective device registered to the user. Each user profile may include skill identifiers of skills125that the user has enabled. When a user enables a skill125, the user is providing the system120with permission to allow the skill125to execute with respect to the user's natural language inputs. If a user does not enable a skill125, the system120may not execute the skill125with respect to the user's natural language inputs.

The profile storage170may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a household profile that is associated with user profiles associated with multiple users of a single household. A group profile may include preferences shared by all the user profiles associated therewith. Each user profile associated with a group profile may additionally include preferences specific to the user associated therewith. That is, a user profile may include preferences unique from one or more other user profiles associated with the same group profile. A user profile may be a stand-alone profile or may be associated with a group profile. A group profile may be associated with (or include) one or more device profiles corresponding to one or more devices associated with the group profile.

The profile storage170may include one or more device profiles. Each device profile may be associated with a different device identifier. A device profile may include various device identifying data, input/output characteristics, networking characteristics, etc. A device profile may also include one or more user identifiers, corresponding to one or more user profiles associated with the device profile. For example, a household device's profile may include the user identifiers of users of the household.

The foregoing describes illustrative components and processing of the system120. The following describes illustrative components and processing of the device110. As illustrated inFIG.3, in at least some embodiments the system120may receive audio data311from the device110, to recognize speech corresponding to a spoken natural language in the received audio data311, and to perform functions in response to the recognized speech. In at least some embodiments, these functions involve sending directives (e.g., commands), from the system120to the device110to cause the device110to perform an action, such as output synthesized speech (responsive to the spoken natural language input) via a loudspeaker(s), and/or control one or more secondary devices by sending control commands to the one or more secondary devices.

Thus, when the device110is able to communicate with the system120over the network(s)199, some or all of the functions capable of being performed by the system120may be performed by sending one or more directives over the network(s)199to the device110, which, in turn, may process the directive(s) and perform one or more corresponding actions. For example, the system120, using a remote directive that is included in response data (e.g., a remote response), may instruct the device110to output synthesized speech via a loudspeaker(s) of (or otherwise associated with) the device110, to output content (e.g., music) via the loudspeaker(s) of (or otherwise associated with) the device110, to display content on a display of (or otherwise associated with) the device110, and/or to send a directive to a secondary device (e.g., a directive to turn on a smart light). It will be appreciated that the system120may be configured to provide other functions in addition to those discussed herein, such as, without limitation, providing step-by-step directions for navigating from an origin location to a destination location, conducting an electronic commerce transaction on behalf of the user105as part of a shopping function, establishing a communication session (e.g., an audio or video call) between the user105and another user, and so on.

The device110may include a wakeword detection component320configured to detect a wakeword (e.g., “Alexa”) that indicates to the device110that the audio data311is to be processed for determining NLU output data. In at least some embodiments, a hybrid selector324, of the device110, may send the audio data311to the wakeword detection component320. If the wakeword detection component320detects a wakeword in the audio data311, the wakeword detection component320may send an indication of such detection to the hybrid selector324. In response to receiving the indication, the hybrid selector324may send the audio data311to the system120and/or an on-device ASR component150. The wakeword detection component320may also send an indication, to the hybrid selector324, representing a wakeword was not detected. In response to receiving such an indication, the hybrid selector324may refrain from sending the audio data311to the system120, and may prevent the on-device ASR component150from processing the audio data311. In this situation, the audio data311can be discarded.

The device110may conduct its own speech processing using on-device language processing components (such as an on-device SLU component, an on-device ASR component350, and/or an on-device NLU component360) similar to the manner discussed above with respect to the speech processing system-implemented ASR component150, and NLU component160. The device110may also internally include, or otherwise have access to, other components such as one or more skills190, a user recognition component395(configured to process in a similar manner to the user recognition component195), profile storage370(configured to store similar profile data to the profile storage170), a TTS component380(configured to process in a similar manner to the TTS component180), one or more skill component(s)390(configured to process in a similar manner to the skill component(s)190), and other components. In at least some embodiments, the on-device profile storage370may only store profile data for a user or group of users specifically associated with the device110.

In at least some embodiments, the on-device language processing components may not have the same capabilities as the language processing components implemented by the system120. For example, the on-device language processing components may be configured to handle only a subset of the natural language inputs that may be handled by the speech processing system-implemented language processing components. For example, such subset of natural language inputs may correspond to local-type natural language inputs, such as those controlling devices or components associated with a user's home. In such circumstances the on-device language processing components may be able to more quickly interpret and respond to a local-type natural language input, for example, than processing that involves the system120. If the device110attempts to process a natural language input for which the on-device language processing components are not necessarily best suited, the NLU output data, determined by the on-device components, may have a low confidence or other metric indicating that the processing by the on-device language processing components may not be as accurate as the processing done by the system120.

The hybrid selector324, of the device110, may include a hybrid proxy (HP)326configured to proxy traffic to/from the system120. For example, the HP326may be configured to send messages to/from a hybrid execution controller (HEC)327of the hybrid selector324. For example, command/directive data received from the system120can be sent to the HEC327using the HP326. The HP326may also be configured to allow the audio data311to pass to the system120while also receiving (e.g., intercepting) this audio data311and sending the audio data311to the HEC327.

In at least some embodiments, the hybrid selector324may further include a local request orchestrator (LRO)328configured to notify the on-device ASR component150about the availability of the audio data311, and to otherwise initiate the operations of on-device language processing when the audio data311becomes available. In general, the hybrid selector324may control execution of on-device language processing, such as by sending “execute” and “terminate” events/instructions. An “execute” event may instruct a component to continue any suspended execution (e.g., by instructing the component to execute on a previously-determined intent in order to determine a directive). Meanwhile, a “terminate” event may instruct a component to terminate further execution, such as when the device110receives directive data from the system120and chooses to use that remotely-determined directive data.

Thus, when the audio data311is received, the HP326may allow the audio data311to pass through to the system120and the HP326may also input the audio data311to the on-device ASR component150by routing the audio data311through the HEC327of the hybrid selector324, whereby the LRO328notifies the on-device ASR component150of the audio data311. At this point, the hybrid selector324may wait for response data from either or both the system120and/or the on-device language processing components. However, the disclosure is not limited thereto, and in some examples the hybrid selector324may send the audio data311only to the on-device ASR component150without departing from the disclosure. For example, the device110may process the audio data311on-device without sending the audio data311to the system120.

The on-device ASR component350is configured to receive the audio data311from the hybrid selector324, and to recognize speech in the audio data311, and the on-device NLU component360is configured to determine an intent from the recognized speech (an optionally one or more named entities), and to determine how to act on the intent by generating NLU output data that may include directive data (e.g., instructing a component to perform an action). In some cases, a directive may include a description of the intent (e.g., an intent to turn off {device A}). In some cases, a directive may include (e.g., encode) an identifier of a second device(s), such as kitchen lights, and an operation to be performed at the second device(s). Directive data may be formatted using Java, such as JavaScript syntax, or JavaScript-based syntax. This may include formatting the directive using JSON. In at least some embodiments, a device-determined directive may be serialized, much like how remotely-determined directives may be serialized for transmission in data packets over the network(s)199. In at least some embodiments, a device-determined directive may be formatted as a programmatic application programming interface (API) call with a same logical operation as a remotely-determined directive. In other words, a device-determined directive may mimic a remotely-determined directive by using a same, or a similar, format as the remotely-determined directive.

A NLU hypothesis (output by the on-device NLU component360) may be selected as usable to respond to a natural language input, and local response data may be sent (e.g., local NLU output data, local knowledge base information, internet search results, and/or local directive data) to the hybrid selector324, such as a “ReadyToExecute” response. The hybrid selector324may then determine whether to use directive data from the on-device components to respond to the natural language input, to use directive data received from the system120, assuming a remote response is even received (e.g., when the device110is able to access the system120over the network(s)199), or to determine output data requesting additional information from the user105.

The device110and/or the system120may associate a unique identifier with each natural language input. The device110may include the unique identifier when sending the audio data311to the system120, and the response data from the system120may include the unique identifier to identify to which natural language input the response data corresponds.

In at least some embodiments, the device110may include one or more skill components190. The skill component(s)190installed on (or in communication with) the device110may include, without limitation, a smart home skill and/or a device control skill configured to control a second device(s), a music skill configured to output music, a navigation skill configured to output directions, a shopping skill configured to conduct an electronic purchase, and/or the like.

In order to apply machine learning techniques, machine learning processes themselves need to be trained. Training a machine learning model requires establishing a “ground truth” for the training examples. In machine learning, the term “ground truth” refers to the accuracy of a training set's classification for supervised learning techniques. Various techniques may be used to train the models including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

As used herein, a “domain” may refer to a collection of related functionality. A domain may be associated with one or more skills performing related functionality may be part of a domain. A non-limiting list of domains includes a smart home domain (corresponding to smart home functionality), a music domain (corresponding to music functionality), a video domain (corresponding to video functionality), a weather domain (corresponding to weather functionality), a communications domain (corresponding to one- or two-way communications functionality), and a shopping domain (corresponding to shopping functionality).

Referring now toFIG.4, it is described how NLU processing may be performed. The NLU component160/360may include one or more recognizers463. In at least some embodiments, a recognizer463may be associated with a skill component190/390(e.g., the recognizer may be configured to interpret a natural language input to correspond to the skill component190/390). In at least some other examples, a recognizer463may be associated with a domain (e.g., the recognizer may be configured to interpret a natural language input to correspond to the domain). A non-limiting list of domains includes a smart home domain, a music domain, a video domain, a flash briefing domain, a shopping domain, a communications domain, and/or a custom domain.

Recognizers463may process in parallel, in series, partially in parallel, etc. For example, a recognizer corresponding to a first domain may process at least partially in parallel to a recognizer corresponding to a second domain. For further example, a recognizer corresponding to a first skill component may process at least partially in parallel to a recognizer corresponding to a second skill component. In another example, a recognizer corresponding to a domain may process at least partially in parallel to a recognizer corresponding to a skill component.

The NLU component160/360may communicate with various storages. The NLU component160/360may communicate with an NLU storage, which includes skill component grammars, representing how natural language inputs may be formulated to invoke skill components190/390, and skill component intents representing intents supported by respective skill components190/390.

Each recognizer463may be associated with a particular grammar, one or more particular intents, and a particular personalized lexicon (stored in an entity library). A gazetteer may include skill component-indexed lexical information associated with a particular user. For example, Gazetteer A may include skill component-indexed lexical information. A user's music skill component lexical information might include album titles, artist names, and song names, for example, whereas a user's contact list skill component lexical information might include the names of contacts. Since every user's music collection and contact list is presumably different, this personalized information may improve entity resolution.

Each recognizer463may include a named entity recognition (NER) component462that attempts to identify grammars and lexical information that may be used to construe meaning with respect to text data input therein. A NER component462identifies portions of text data that correspond to a named entity that may be recognizable by the system120/device110. A NER component462may also determine whether a word refers to an entity that is not explicitly mentioned in the text, for example “him,” “her,” “it” or other anaphora, exophora or the like.

A NER component462applies grammar models and lexical information associated with one or more skill components190/390to determine a mention of one or more entities in text data input therein. In this manner, a NER component462identifies “slots” (i.e., particular words in text data) that may be needed for later processing. A NER component462may also label each slot with a type (e.g., noun, place, city, artist name, song name, etc.).

Each grammar model may include the names of entities (i.e., nouns) commonly found in natural language about a particular skill component190/390to which the grammar model relates, whereas lexical information may be personalized to the user identifier output by a user recognition component190/390for the natural language input. For example, a grammar model associated with a shopping skill component may include a database of words commonly used when people discuss shopping.

A downstream process called named entity resolution links a portion of input data (identified by a NER component462) to a specific entity known to the system120/device110. To perform named entity resolution, the NLU component160/360may use gazetteer information stored in the entity library storage. The gazetteer information may be used to match text data (identified by a NER component462) with different entities, such as song titles, contact names, etc. Gazetteers may be linked to users (e.g., a particular gazetteer may be associated with a specific user's music collection), may be linked to certain skill components190/390(e.g., a shopping skill component, a music skill component, a video skill component, a communications skill component, etc.), or may be organized in another manner.

Each recognizer463may also include an intent classification (IC) component464that processes text data input thereto to determine an intent(s) of a skill component(s)190/390that potentially corresponds to the natural language input represented in the text data. An intent corresponds to an action to be performed that is responsive to the natural language input represented by the text data. An IC component464may communicate with a database of words linked to intents. For example, a music intent database may link words and phrases such as “quiet,” “volume off,” and “mute” to a <Mute> intent. An IC component464identifies potential intents by comparing words and phrases in text data to the words and phrases in an intents database associated with the skill component(s)190/390that is associated with the recognizer463implementing the IC component464.

The intents identifiable by a specific IC component464may be linked to one or more skill component-specific grammar frameworks with “slots” to be filled. Each slot of a grammar framework corresponds to a portion of text data that a NER component462believes corresponds to an entity. For example, a grammar framework corresponding to a <PlayMusic> intent may correspond to text data sentence structures such as “Play {Artist Name},” “Play {Album Name},” “Play {Song name},” “Play {Song name} by {Artist Name},” etc. However, to make resolution more flexible, grammar frameworks may not be structured as sentences, but rather based on associating slots with grammatical tags.

For example, a NER component462may identify words in text data as subject, object, verb, preposition, etc. based on grammar rules and/or models prior to recognizing named entities in the text data. An IC component464(implemented by the same recognizer463) may use the identified verb to identify an intent. The NER component462may then determine a grammar model associated with the identified intent. For example, a grammar model for an intent corresponding to <PlayMusic> may specify a list of slots applicable to play the identified “object” and any object modifier (e.g., a prepositional phrase), such as {Artist Name}, {Album Name}, {Song name}, etc. The NER component462may then search corresponding fields in a lexicon, attempting to match words and phrases in the text data the NER component462previously tagged as a grammatical object or object modifier with those identified in the lexicon.

A NER component462may perform semantic tagging, which is the labeling of a word or combination of words according to their type/semantic meaning. A NER component462may parse text data using heuristic grammar rules, or a model may be constructed using techniques such as hidden Markov models, maximum entropy models, log linear models, conditional random fields (CRF), and the like. For example, a NER component462, implemented by a music skill component or music domain recognizer463, may parse and tag text data corresponding to “play mother's little helper by the rolling stones” as {Verb}: “Play,” {Object}: “mother's little helper,” {Object Preposition}: “by,” and {Object Modifier}: “the rolling stones.” The NER component462may identify “Play” as a verb based on a word database associated with the music skill component or music domain, which an IC component464may determine corresponds to a <PlayMusic> intent. At this stage, no determination has been made as to the meaning of “mother's little helper” and “the rolling stones,” but based on grammar rules and models, the NER component462has determined that the text of these phrases relates to the grammatical object (i.e., entity).

The frameworks linked to the intent are then used to determine what database fields should be searched to determine the meaning of these phrases, such as searching a user's gazetteer for similarity with the framework slots. For example, a framework for a <PlayMusic> intent might indicate to attempt to resolve the identified object based on {Artist Name}, {Album Name}, and {Song name}, and another framework for the same intent might indicate to attempt to resolve the object modifier based on {Artist Name}, and resolve the object based on {Album Name} and {Song Name} linked to the identified {Artist Name}. If the search of the gazetteer does not resolve a slot/field using gazetteer information, the NER component462may search a database of generic words (in a knowledge base). For example, if the text data corresponds to “play songs by the rolling stones,” after failing to determine an album name or song name called “songs” by “the rolling stones,” the NER component462may search a music skill component vocabulary for the word “songs.” In the alternative, generic words may be checked before the gazetteer information, or both may be tried, potentially producing two different results.

As described above, more than one recognizer463may process with respect to ASR data402representing a single natural language input. In such instances, each recognizer463may output at least one NLU hypothesis including an intent indicator (determined by an IC component464of the recognizer463) and at least one tagged named entity (determined by a NER component462of the recognizer463).

In some embodiments, the recognizer463, including the NER component462and the IC component464, may process with respect to the widget context data114. The IC component464may use the information included in the widget context data114to determine intent data corresponding to a user input. The NER component462may use the information included in the widget context data114to determine entity data.

The NLU component160/360may include a shortlister component410. The shortlister component410selects skill components190/390that may execute in response to the natural language input. The shortlister component410thus limits downstream, more resource intensive NLU processes to being performed with respect to skill components190/390that are likely to execute in response to the natural language input.

Without a shortlister component410, the NLU component160/360may process a given ASR hypothesis, included in the ASR data402, (or the text data depending on the type of natural language input being processed) with respect to every skill component190/390of (or in communication with) the system120, either in parallel, in series, or using some combination thereof. By implementing a shortlister component410, the NLU component160/360may process a given ASR hypothesis included in the ASR data402with respect to only the skill components190/390that are likely to execute in response to the natural language input. This reduces total compute power and latency attributed to NLU processing.

The shortlister component410may include one or more machine learning (ML) models. The ML model(s) may be trained to recognize various forms of natural language inputs that may be received by the system120/device110. For example, during a training period a skill component190/390may provide the system120/device110with training data representing sample natural language inputs that may be provided by a user to invoke the skill component190/390. For example, a ride sharing skill component may provide the system120/device110with training data including text corresponding to “get me a cab to [location],” “get me a ride to [location],” “book me a cab to [location],” “book me a ride to [location],” etc. The one or more ML models may be trained using the training data to determine other potentially related natural language input structures that a user may try to use to invoke the particular skill component190/390. During training, the system120/device110may query the skill component190/390regarding whether the determined other natural language input structures are permissible, from the perspective of the skill component190/390, to be used to invoke the skill component190/390at runtime. The alternate natural language input structures may be derived by one or more ML models during model training and/or may be based on natural language input structures provided by different skill components190/390. The skill component190/390may also provide the system120/device110with training data indicating grammar and annotations. The system120/device110may use the training data representing the sample natural language inputs, the determined related natural language input(s), the grammar, and the annotations to train a ML model that indicates when a runtime natural language input is likely to be directed to/handled by a particular skill component190/390. Each ML model of the shortlister component410may be trained with respect to a different skill component190/390. Alternatively, the shortlister component410may implement one ML model per skill component type, such as one ML model for weather skill components, one ML model for ride sharing skill components, etc.

The system120/device110may use the sample natural language inputs provided by a skill component190/390, and related sample natural language inputs determined during training, as binary examples to train a ML model associated with the skill component190/390. The ML model associated with the particular skill component190/390may then be operated at runtime by the shortlister component410. Some sample natural language inputs may be positive training examples (e.g., natural language inputs that may be used to invoke the skill component190/390), whereas other sample natural language inputs may be negative training examples (e.g., natural language inputs that may not be used to invoke the skill component190/390).

As described above, the shortlister component410may include a different ML model for each skill component190/390, a different ML model for each skill component type, or some other combination of ML models. For example, the shortlister component410may alternatively include a single ML model. The single ML model may include a portion trained with respect to characteristics (e.g., semantic characteristics) shared by all skill components190/390. The single ML model may also include skill component-specific portions, with each skill component-specific portion being trained with respect to a different skill component190/390. Implementing a single ML model with skill component-specific portions may result in less latency than implementing a different ML model for each skill component190/390because the single ML model with skill component-specific portions limits the number of characteristics processed on a per skill component level.

The portion of the ML model, trained with respect to characteristics shared by more than one skill component190/390, may be clustered based on skill component type. For example, a first portion, of the portion trained with respect to multiple skill components190/390, may be trained with respect to weather skill components; a second portion, of the portion trained with respect to multiple skill components190/390, may be trained with respect to music skill components; a third portion, of the portion trained with respect to multiple skill components190/390, may be trained with respect to travel skill components; etc.

Clustering may not be beneficial in every instance because clustering may cause the shortlister component410to output indications of only a portion of the skill components190/390that the natural language input may relate to. For example, a natural language input may correspond to “tell me about Tom Collins.” If the ML model is clustered based on skill component type, the shortlister component410may determine the natural language input corresponds to a recipe skill component (e.g., storing or otherwise having access to a drink recipe) even though the natural language input may also correspond to an information skill component (e.g., storing or otherwise having access to information about a person named Tom Collins).

Training the ML model(s) of the shortlister component410may require establishing a “ground truth” for the training examples input therein. In machine learning, the term “ground truth” refers to the accuracy of a training set's classification for supervised learning techniques. Various techniques may be used to train the models including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

If the shortlister component410determines a natural language input is associated with multiple skill components190/390, only the recognizers463associated with those skill components190/390may process with respect to the natural language input. The selected recognizers463may process in parallel, in series, partially in parallel, etc. For example, if the shortlister component410determines a natural language input may relate to both a communications skill component and a music skill component, a recognizer463associated with the communications skill component may process in parallel, or partially in parallel, with a recognizer463associated with the music skill component processing.

The shortlister component410may make binary determinations (e.g., yes or no) regarding which skill component(s)190/390corresponds to a natural language input. The shortlister component410may make such determinations using the one or more ML models described herein above. If the shortlister component410implements a single ML model for each skill component190/390, the shortlister component410may simply run the ML models that are associated with enabled skill components190/390as indicated in a profile (e.g., stored in the profile storage170/370) associated with the device110and/or user105that originated the natural language input.

In some embodiments, the shortlister component410may use the widget context data114to determine the skill components190/390that correspond to the user input. The shortlister component410may process the widget context data114using one or more of the ML models described above.

The shortlister component410may generate shortlisted skill component data415representing one or more skill components190/390that may execute in response to the user input. The skill components190/390included in the shortlisted skill component data415may be based on the widget context data114. The number of skill components190/390represented in the shortlisted skill component data415is configurable. In an example, the shortlisted skill component data415may indicate every skill component190/390of (or otherwise in communication with) the system120/device110as well as contain an indication, for each skill component190/390, representing whether the skill component190/390is likely capable of processing in response to the natural language input. In another example, instead of indicating every skill component190/390, the shortlisted skill component data415may only indicate the skill components190/390that are likely capable of processing in response to the natural language input. In yet another example, the shortlister component410may implement thresholding such that the shortlisted skill component data415may indicate no more than a maximum number of skill components190/390that may process in response to the natural language input.

In at least some embodiments, the shortlister component410may generate a score representing how likely a skill component190/390is likely to process in response to a user input. In such embodiments, the shortlisted skill component data415may only include identifiers of skill components190/390associated with scores meeting or exceeding a threshold score.

In the situation where the ASR component150/350outputs ASR data402including more than one interpretation of a spoken input, the shortlister component410may output different shortlisted skill component data415for each interpretation. Alternatively, the shortlister component410may output a single shortlisted skill component data415representing the skill components190/390corresponding to the different interpretations.

As indicated above, the shortlister component410may implement thresholding such that the shortlisted skill component data415may indicate no more than a threshold number of skill components190/390(e.g., may include no more than a threshold number of skill component identifiers). If the ASR component150/350outputs ASR data402including more than one interpretation of a natural language input, the shortlisted skill component data415may indicate no more than a threshold number of skill components190/390irrespective of the number of interpretations output by the ASR component150/350. Alternatively or in addition, the shortlisted skill component data415may indicate no more than a threshold number of skill components190/390for each interpretation (e.g., indicating no more than five skill components190/390for a first interpretation, no more than five skill components190/390for a second interpretation, etc.).

In addition to making a binary determination regarding whether a skill component190/390may process in response to a natural language input, the shortlister component410may generate scores representing likelihoods that skill components190/390may process in response to the natural language input. If the shortlister component410implements a different ML model for each skill component190/390, the shortlister component410may generate a different confidence score for each skill component-specific ML model that is run by the shortlister component410. For example, if the shortlister component410runs the ML models of every skill component190/390of (or otherwise in communication with) the system120/device110, the shortlister component410may generate a respective confidence score for each skill component190/390. For further example, if the shortlister component410only runs ML models specific to skill components190/390that are indicated as enabled in a profile associated with the device110and/or user105(as stored in the profile storage170/370), the shortlister component410may only generate a respective confidence score for each enabled skill component190/390. For further example, if the shortlister component410implements a single ML model with skill component-specific portions, the shortlister component410generate a respective confidence score for each skill component190/390who's specifically trained portion is run. The shortlister component410may perform matrix vector modification to obtain confidence scores for skill components190/390.

An example of shortlisted skill component data415including confidence scores may be represented as:Search skill component, 0.67Recipe skill component, 0.62Information skill component, 0.57
As indicated, the confidence scores output by the shortlister component410may be numeric values. The confidence scores output by the shortlister component410may alternatively be binned values (e.g., high, medium, low).

The shortlister component410may consider other data420when determining which skill components190/390may process in response to a natural language input. The other data420may be character embedded prior to being input to the shortlister component410. The other data420may alternatively be embedded using other techniques.

The other data420may include usage history data associated with the device110and/or user105. For example, a confidence score of a skill component190/390may be increased if natural language inputs captured by the device110and/or originating from the user105routinely relate to the skill component190/390. Conversely, a confidence score of a skill component190/390may be decreased if natural language inputs captured by the device110and/or originated from the user105rarely relate to the skill component190/390.

The other data420may indicate the skill components190/390that are enabled with respect to the device110and/or user105(e.g., as represented in the profile storage170/370). The shortlister component410may use such data to determine which skill component-specific ML models to run. That is, the shortlister component410may determine to only run the ML models associated with enabled skill components190/390. The shortlister component410may alternatively use such data to alter skill component confidence scores represented in the shortlisted skill component data415. As an example, considering two skill components190/390, one enabled and another unenabled, the shortlister component410may run a first ML model (or ML model portion) specific to the unenabled skill component as well as a second ML model (or ML model portion) specific to the enabled skill component. The shortlister component410may initially determine a confidence score of 0.60 for each of the unenabled skill component and the enabled skill component. The shortlister component410may then alter those confidence scores based on which skill component is enabled. For example, the shortlister component410may increase the confidence score associated with the enabled skill component and/or decrease the confidence score associated with the unenabled skill component.

A user105may provide the system120with indications of which skill components190/390are enabled (e.g., authorized to execute using data associated with the user105). Such indications may be stored in the profile storage170/370. The shortlister component410may determine whether profile data associated with the user105and/or device110that originated the natural language input includes indications of enabled skill components190/390.

The other data420may indicate a type of the device110. The type of the device110may indicate the input/output capabilities of the device110. For example, the device110may include a display, may be headless (e.g., displayless), may be mobile or stationary, may include audio playback capabilities, may include a camera, etc. The shortlister component410may use such other data420to determine which skill component-specific ML models (or ML model portions) to run. For example, if the device110is displayless, the shortlister component410may determine not to run ML models (or ML model portions) specific to skill components190/390that output video data. The shortlister component410may alternatively use such other data420to alter skill component confidence scores represented in the shortlisted skill component data415. As an example, considering two skill components190/390, one that outputs audio data and another that outputs video data, the shortlister component410may run a first ML model (or portion of a ML model) specific to the skill component that generates audio data as well as a second ML model (or portion of a ML model) specific to the skill component that generates video data. The shortlister component410may initially determine a confidence score of 0.60 for each of the skill components. The shortlister component410may then alter the original confidence scores based on the type of the device110. For example, if the device110is displayless, the shortlister component410may increase the confidence score associated with the skill component that generates audio data and/or decrease the confidence score associated with the skill component that generates video data.

The type of device data represented in the other data420may represent output capabilities of the device110to be used to output content to the user105, which may not necessarily be the device110that captured the natural language input. For example, a displayless device110may receive a natural language input corresponding to “play Game of Thrones”. The system120/device110may determine a smart TV, or other device including or otherwise associated with a display, is to be used to output multimedia content (e.g., audio and video) corresponding to the title “Game of Thrones”.

The other data420may include data indicating a speed of the device110, a location of the device110, or other mobility data. For example, the device110may correspond to a vehicle including a display. If the vehicle is moving, the shortlister component410may decrease the confidence score associated with a skill component190/390that generates video data as it may be undesirable to output video content to the user105while in a moving vehicle. The vehicle may output data to the system120indicating when the vehicle is in motion.

The other data420may include data indicating a currently invoked skill component190/390(e.g., a skill component190/390that was processing to output content to the user105when the device110received the natural language input). For example, the user105may speak a first (e.g., a previous) natural language input causing the system120to invoke a music skill component to output music to the user105. As the music is being output to the user105, the system120may receive a second (e.g., a current) natural language input. The shortlister component410may use such other data420to alter skill component confidence scores represented in the shortlisted skill component data415. For example, the shortlister component410may run a first ML model (or ML model portion) specific to a first skill component as well as a second ML model (or ML model portion) specific to a second skill component. The shortlister component410may initially determine a confidence score of 0.60 for each of the first and second skill components. The shortlister component410may then alter the original confidence scores based on the first skill component being invoked to output content while the current natural language input was received. Based on the first skill component being invoked, the shortlister component410may increase the confidence score associated with the first skill component and/or decrease the confidence score associated with the second skill component.

The thresholding implemented with respect to the shortlisted skill component data415and the different types of other data420considered by the shortlister component410are configurable. For example, the shortlister component410may update confidence scores as more other data420is considered.

The shortlister component410may cause the NLU component160/360to invoke only a subset of the recognizers463associated with skill components190/390represented in the shortlisted skill component415as being likely to process in response to the natural language input. If the shortlister component410generates the shortlisted skill component data415to include confidence scores, the shortlister component410may cause the NLU component160/360to invoke only recognizers463associated with skill components190/390associated with confidence scores satisfying (e.g., meeting or exceeding) a condition (e.g., a threshold confidence score).

The NLU component160/360may compile NLU hypotheses, output by multiple recognizers463, into cross-recognizer hypothesis data440(illustrated inFIG.4). Each NLU hypothesis may be associated with a respective score indicating a likelihood that the NLU hypothesis corresponds to the domain, one or more skill components190/390, etc. associated with the recognizer463from which the NLU hypothesis was output. For example, the cross-recognizer hypothesis data440may be represented as, with each line corresponding to a different NLU hypothesis:[0.95] Intent: <PlayMusic> AlbumName: GameOfThrones[0.70] Intent: <PlayVideo> VideoTitle: GameOfThrones.
While the foregoing illustrates cross-recognizer hypothesis data440include two NLU hypotheses, it will be appreciated that the cross-recognizer hypothesis data440may include differing numbers of NLU hypotheses with departing from the present disclosure.

The NLU component160/360may send the cross-recognizer hypothesis data440to a pruning component450, which sorts the NLU hypotheses, represented in the cross-recognizer hypothesis data440, according to their respective scores. The pruning component450may then perform score thresholding with respect to the cross-recognizer hypothesis data440. For example, the pruning component450may select NLU hypotheses, represented in the cross-recognizer hypothesis data440, associated with scores satisfying (e.g., meeting and/or exceeding) a condition (e.g., a threshold score). The pruning component450may additionally or alternatively perform number of NLU hypothesis thresholding. For example, the pruning component450may select a threshold number of top-scoring NLU hypotheses represented in the cross-recognizer hypothesis data440.

The pruning component450may generate cross-recognizer hypothesis data460including the selected NLU hypotheses. The purpose of the pruning component450is to create a reduced list of NLU hypotheses so that downstream, more resource intensive, processes may only operate on NLU hypotheses that most likely represent the natural language input.

The NLU component160/360may include a light slot filler component452that takes text from slots, represented in the NLU hypotheses output by the pruning component450, and alters it to make the text more easily processed by downstream components. The light slot filler component452may perform low latency operations that do not involve heavy operations such as reference to a knowledge base. The purpose of the light slot filler component452is to replace words with other words or values that may be more easily understood by downstream components. For example, if a NLU hypothesis includes the word “tomorrow,” the light slot filler component452may replace the word “tomorrow” with an actual date for purposes of downstream processing. Similarly, the light slot filler component452may replace the word “CD” with “album” or “compact disc.” The replaced text is then included in the cross-recognizer hypothesis data460.

The cross-recognizer hypothesis data460may be sent to an entity resolution component470. The entity resolution component470can apply rules or other instructions to standardize labels or tokens in the NLU hypotheses represented in the cross-recognizer hypothesis data460. The precise transformation may depend on the skill component190/390, domain, etc. to which a NLU hypothesis relates. For example, for a travel skill component-specific NLU hypothesis, the entity resolution component470may transform text corresponding to “Boston airport” to the standard “BOS” three-letter code referring to the airport. The entity resolution component470can refer to a knowledge base that is used to specifically identify the precise entity referred to in each slot of each NLU hypothesis represented in the cross-recognizer hypothesis data460.

Specific intent/slot combinations may also be tied to a particular source, which may then be used to resolve the text. Referring to the example natural language input “play songs by the stones,” the entity resolution component470may reference a personal music catalog, a user profile, or the like (for example stored in the profile storage170/370). The entity resolution component470may output NLU hypotheses, altered from the cross-recognizer hypothesis data460, that include more detailed information (e.g., entity identifiers) about the specific entities mentioned in the slots and/or more detailed slot data that can eventually be used by a skill component190/390to perform an action responsive to the natural language input. The NLU component160/360may include multiple entity resolution components470that are each specific to one or more different skill components190/390, domains, etc.

In some embodiments, the entity resolution component470may use the widget context data114to determine the specific entity corresponding to a slot/entity represented in the user input.

The NLU component160/360may include a ranker component490that assigns a particular score to each NLU hypothesis output by the entity resolution component470. The ranker component490may alter the score of a particular NLU hypothesis based on whether the NLU hypothesis has unfilled slots. For example, if a first NLU hypothesis includes slots that are all filled/resolved, the ranker component490may assign the first NLU hypothesis a higher score than a second NLU hypothesis including at least one slot that is unfilled/unresolved by the entity resolution component470.

The ranker component490may apply re-scoring, biasing, or other techniques. To do so, the ranker component490may consider not only the data output by the entity resolution component470, but may also other data491. The other data491may include a variety of data.

For example, the other data491indicate skill component190/390rating or popularity. For example, if a skill component190/390has a high rating, the ranker component490may increase the score of a NLU hypothesis associated with that skill component190/390, and vice versa.

The other data491may indicate skill components190/390that have been enabled by the user105. For example, the ranker component490may assign higher scores to NLU hypotheses associated with enabled skill components than NLU hypotheses associated with skill components that have not been enabled by the user105.

The other data491include system usage history (e.g., specific to the user105), such as if the user105regularly invokes a particular skill component190/390or does so at particular times of day. The other data491may indicate a present date, a present time, location of the device110, weather data, a type of the device110, a user identifier of the user105, context data, as well as other data. For example, the ranker component490may consider when any particular skill component190/390is currently active with respect to the present user105and/or device110(e.g., music being output by the skill component190/390when the current natural language input is received).

The ranker component490may output NLU output data485including one or more NLU hypotheses. The NLU component160/360may send the NLU output data485to the orchestrator component130.

Referring toFIG.5, details of how the skill selection component165/365may be configured are described. When the skill selection component165/365receives NLU output data485from the NLU component160, the NLU output data485may be sent to a skill pre-response component510. The skill pre-response component510is configured to query skills, associated with the NLU output data485as to whether the skills are able to process the NLU hypothesis with which they are associated. In some embodiments, the skill pre-response component510may query skills associated with the multi-modal widgets112as to whether the skills are able to process an NLU hypothesis.

The skill pre-response component510sends a pre-response query to each skill component190/390associated with a NLU hypothesis represented in the NLU output data485. A pre-response query may include a NLU hypothesis (associated with the skill component190/390receiving the pre-response query), portion of the widget context data114that corresponds to the queried skill, and optionally other context data corresponding to the user input.

A skill component190/390may determine, based on a received pre-response query (and optionally other data available to the skill component190/390), whether the skill190/390could respond to the natural language input. For example, a skill component190/390may generate a pre-response indicating the skill component190/390can respond to the natural language input, indicating the skill component190/390may be able to respond to the natural language input (e.g., the indicating the skill component190/390needs more data to determine whether the skill component190/390can respond to the natural language input), or indicating the skill component190/390cannot respond to the natural language input (e.g., due to present processing load).

In situations where a skill's pre-response indicates the skill component190/390can or may be able to respond to the natural language input, the pre-response may also include various other data representing a strength of the skill's potential response to the natural language input. Such other data may positively influence the skill's ranking as described herein below. For example, such other data may indicate capabilities (e.g., output capabilities or components such as a connected screen, loudspeaker, etc.) of a device to be used to output the skill's response; pricing data corresponding to a product or service the natural language input is requesting be purchased (or is requesting purchase data for); availability of a product the natural language input is requesting be purchased; whether there are shipping fees for a product the natural language input is requesting be purchased; whether the user105already has a profile and/or subscription with the skill component190/390; that the user105does not have a subscription with the skill component190/390, but that there is a free trial/tier the skill component190/390is offering; with respect to a taxi skill, a cost of a trip based on start and end locations, how long the user105would have to wait to be picked up, etc.; and/or other data available to the skill component190/390that is related to the skill's processing of the natural language input. In at least some embodiments, a skill's pre-response may include an indicator (e.g., a flag) representing a strength of the skill's ability to personalize its response to the user105.

In at least some embodiments, a skill's pre-response may be configured to a schema pre-defined by the system120. By the system120requiring pre-responses to conform to a specific schema (e.g., by requiring skill components190/390to only be able to provide certain types of data in pre-responses), the system120may onboard new skill components190/390into the skill selection functionality described herein without needing to reconfigure the skill selection component165/365each time a new skill component190/390is onboarded. Moreover, requiring pre-responses to conform to a schema limits the amount of values needed to be used to train and implement a machine learning model for ranking skill components190/390, as described herein below.

The skill pre-response component510may receive pre-response data from the queried skills, and output data515including the skill identifiers associated with respective pre-responses. As indicated above, a skill's pre-response data may indicate the skill component190/390would be unable to process if called to execute with respect to the natural language input. In at least some embodiments, the skill pre-response component510may only include, in the data515, the skill identifiers associated with pre-response data that does not indicate an associated skill component190/390would be unable to process. The data515, output by the skill pre-response component510, may not include any confidence scores representing associated skill component190/390should be used to respond to the natural language input.

The data515, output by the skill pre-response component510, may be input to a skill ranker520of the skill selection component165/365. The skill ranker520may be configured to rank skill components190/390using a machine learning model. In at least some embodiments, the machine learning model may be a deep neural network (DNN).

The skill ranker520also receives context data corresponding to the natural language input. Such context data may indicate, for example, one or more preferences (e.g., skill125preferences) of the user105(as represented in a user profile or group profile stored in the profile storage170), one or more domains corresponding to the natural language input, one or more intents representing the natural language input, NER entity values as represented in the NLU output data, a measure of success of entity resolution performed with respect to the natural language input, text representing the natural language input, rankings of individual NLU hypotheses within the NLU output data, a device type of the device110that received the natural language input, whether the device110was outputting audio when the natural language input was received, whether the device110was outputting video when the natural language input was received, and/or other context data available to the skill ranker520. The skill ranker520may also receive the widget context data114.

Using the ML model, the skill ranker520ranks the skill component190/390based on the data515, the widget context data114, and the context data. Things that may increase a skill's ranking include, for example, that the skill component190/390is associated with pre-response data indicating the skill component190/390can generate a response (to the natural language input) that is personalized to the user105, that a NLU hypothesis corresponding to the skill component190/390is associated with a NLU confidence score satisfying (e.g., meeting or exceeding) a condition (e.g., a threshold NLU confidence score), that the skill component190/390corresponds to a music skill and the device110was outputting music when the natural language input was received, that the skill component190/390corresponds to a video skill and the device110was outputting video when the natural language input was received, etc. Things that may decrease a skill's ranking include, for example, that the skill component190/390is associated with pre-response data indicating the skill component190/390cannot generate a response (to the natural language input) that is personalized to the user105, that a NLU hypothesis corresponding to the skill component190/390is associated with a NLU confidence score failing to satisfy (e.g., falling below) a condition (e.g., a threshold NLU confidence score), that the skill component190/390corresponds to a video skill and the device110was outputting music when the natural language input was received, that the skill component190/390corresponds to a music skill and the device110was outputting video when the natural language input was received, etc. Other example are possible and will depend on the context data available to the skill ranker520.

The skill ranker520outputs data525including the skill identifiers associated with respective confidence scores, where a confidence score represents a strength with which the skill ranker520recommends the associated skill component190/390is to execute to respond to the natural language input. Such a confidence score may be a numeric score (e.g., between 0 and 1) or a binned score (e.g., low, medium, high). The data525may also include one or more widget identifiers (and/or widget instance identifiers) corresponding to the multi-modal widgets112, when the skill component190/390included in the data525corresponds to the multi-modal widgets112active or enabled at the device110. As such, the data525may include skill identifiers associated with one or more skill components190/390that has a multi-modal widget active or enabled at the device110, and/or may include skill identifiers associated with one or more skill components190/390that do not have a multi-modal widget active or enabled at the device110.

A decider component530may receive the data525output by the skill ranker520. The decider component530may be configured to generate an execution plan555based on the data525(representing the skill identifiers associated with respective confidence scores) and context data (e.g., representing NLU hypotheses associated with the skill component identifiers and the widget context data114).

In some embodiments, the decider component530may select the skill component190/390associated with the highest confidence score in the data525. In some embodiments, the decider component530may select the skill component190/390associated with a confidence score that satisfies a condition (e.g., exceeds a threshold value). In some embodiments, the decider component530may select the skill component190/390associated with a confidence score that satisfies a threshold condition and satisfies a condition corresponding to context data, such as the widget context data114.

The decider component530may generate an execution plan555including the skill identifier, of the skill component190/390selected by the decider component530to execute in response to the natural language input, the NLU hypothesis associated with the skill identifier, a portion of the widget context data114corresponding to the selected skill component190/390(i.e. the widget context data116) and a command to execute/invoke the skill component190/390corresponding to the skill identifier. The decider component530may send the execution plan555to the skill request router component188, which may cause the skill component190/390(corresponding to the skill identifier in the execution plan555) to process the NLU hypothesis to determine output data responsive to the natural language input.

Multiple systems (120/125) may be included in the system100of the present disclosure, such as, one or more systems120and/or one or more skills125. In operation, each of these systems may include computer-readable and computer-executable instructions that reside on the respective device (120/125), as will be discussed further below.

Each of these devices (110/120/125) may include one or more controllers/processors (604/704), which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (606/706) for storing data and instructions of the respective device. The memories (606/706) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory. Each device (110/120/125) may also include a data storage component (608/708) for storing data and controller/processor-executable instructions. Each data storage component (608/708) may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device (110/120/125) may also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces (602/702).

Computer instructions for operating each device (110/120/125) and its various components may be executed by the respective device's controller(s)/processor(s) (604/704), using the memory (606/706) as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory (606/706), storage (608/708), or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.

Each device (110/120/125) includes input/output device interfaces (602/702). A variety of components may be connected through the input/output device interfaces (602/702), as will be discussed further below. Additionally, each device (110/120/125) may include an address/data bus (624/724) for conveying data among components of the respective device. Each component within a device (110/120/125) may also be directly connected to other components in addition to (or instead of) being connected to other components across the bus (624/724).

Referring toFIG.6, the device110may include input/output device interfaces602that connect to a variety of components such as an audio output component such as a speaker612, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device110may also include an audio capture component. The audio capture component may be, for example, a microphone620or array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. The device110may additionally include a display616for displaying content. The device110may further include a camera618.

The components of the device110, the system120, and/or skill125may include their own dedicated processors, memory, and/or storage. Alternatively, one or more of the components of the device110, the system120, and/or skill125may utilize the I/O interfaces (602/702), processor(s) (604/704), memory (606/706), and/or storage (608/708) of the device110, the system120, and/or skill125, respectively.

As illustrated inFIG.8, multiple devices (110a-110j,120,125) may contain components of the system and the devices may be connected over a network(s)199. The network(s)199may include a local or private network or may include a wide network such as the Internet. Devices may be connected to the network(s)199through either wired or wireless connections. For example, a speech-controllable device110a, a smart phone110b, a smart watch110c, a tablet computer110d, a vehicle110e, a speech-controllable display device110f, a smart television110g, a washer/dryer110h, a refrigerator110i, and/or a microwave110jmay be connected to the network(s)199through a wireless service provider, over a Wi-Fi or cellular network connection, or the like. Other devices are included as network-connected support devices, such as the system120, the skill125, and/or others. The support devices may connect to the network(s)199through a wired connection or wireless connection.