Patent Description:
In the prior art, it is known according to the patent application <CIT>, techniques for processing a spoken input and to modify a provided textual data according to a predefined editing command provided with the spoken input.

It is further known according to the patent application <CIT>, techniques for processing a speech signal to text and determining a next placement within a digital document using a machine learning model by determining that the text is referring to a specific portion of the digital document.

The invention provides a data processing system according to claim <NUM> and a method performed by a data processing system according to claim <NUM>. Preferable aspects are defined by the dependent claims.

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

Techniques for improved voice-based content manipulation are provided herein that use context to enable improved interpretation of voice inputs. The techniques disclosed herein provide a technical solution to the technical problem of differentiating between textual content and commands in voice-based content manipulation scenarios. The techniques disclosed herein utilize one or more natural language processing models to analyze audio inputs received from the user and translate these audio inputs into textual outputs. The audio inputs include textual content, one or more commands to be performed by an application on the textual content, or a combination thereof. The natural language processing models are trained to accurately recognize the words spoken in the audio input and to translate those words into textual output that represents what the user said. The techniques analyze the textual output of the natural language processing models to determine the context of what was said by the user. The first context information of the first textual output provides an indication of whether the first textual output includes the command and an indication of how the user intended to apply the command to content in an application utilizing these techniques. Additional context information may be obtained from the application that may be used to disambiguate the meaning of what was said by the user. The additional context information may include cursor or insertion point (IP) location information and/or other information that indicates how the user was interacting with the application prior to, during, or after issuing a voice command. A technical benefit of the techniques disclosed herein is that the accuracy of the handling of the voice inputs can be significantly improved by anticipating the intent of the user by analyzing the voice inputs using machine learning models trained to recognize numerous ways in which users may interact with the application.

<FIG> is a block diagram illustrating an example computing environment <NUM> in which the techniques disclosed herein may be implemented, and <FIG> is another block diagram of the example computing environment which illustrates example data which may be exchanged between the various components illustrated in the computing environment <NUM>. The computing environment <NUM> includes an application <NUM>, a contextual language processing unit (CLPU) <NUM>, a natural language processing unit (NLPU) <NUM>, a machine learning language processing unit (MLPU) <NUM>, one or more local natural language processing (NLP) models 180a, one or more remote NLP models 180b, one or more local machine learning (ML) models 190a, and one or more remote ML models 190b. In some implementations, the functionality of the one or more of the natural language processing unit <NUM>, the machine learning language processing unit <NUM>, the one or more local NLP models 180a, and the one or more NLP models 180b may be implemented by the contextual language unit <NUM>. Furthermore, in yet other implementations, at least a portion of the functionality of the contextual language processing unit may be implemented by the application <NUM>.

The application <NUM> is configured to receive textual input and/or commands through voice input. The application <NUM> may be a word processing application, an email application, a spreadsheet application, a messaging application, a presentation application, or other type of application that may be configured to receive voice inputs that may include textual input, commands, or both.

The application <NUM> may be implemented on various types of computing devices, such as but not limited to a personal computer (PC), a tablet computer, a laptop computer, a netbook, a gaming and/or entertainment system, a smart phone, a mobile device, or a wearable device. The application <NUM> may be implemented as executable program code implemented on the computing device or may be implemented as a cloud-based application that may be implemented at least in part on one or more remote servers, such as the cloud-based applications <NUM> of <FIG>, and may be accessed through a web-browser or other such interface on the computing device <NUM>.

The CLPU <NUM> utilizes the natural language processing unit <NUM> to provide speech recognition services for converting spoken language content included in an audio sample to text. The CLPU <NUM> utilizes the MLLPU <NUM> to determine a context for the text and to disambiguate the meaning of the text. The CLPU <NUM> is configured to obtain an audio input captured by a microphone or microphones associated with the computing device on which the application <NUM> is being executed and to process that audio input to identify content included in the audio input. The content includes any of textual input to be processed by the application <NUM>, commands to be executed by the application <NUM>, and/or both. The textual input may, for example, be textual content being dictated by the user for inclusion in a document being edited using the application <NUM>. The commands may be related to editing and/or formatting textual input. For example, the user may state "bold the previous word" or "underline that" to render a word of the textual content in a bold font or to underline the word. Other types of formatting commands may be presented by the user in a voice command. The specific types of formatting commands recognized and supported by the application <NUM> depend upon the capabilities of the application <NUM> may be vary from application to application. The content may also include may be related to controlling the operation of one or more features of the application <NUM>. For example, the user may state "save and close document" or "open new document" to save a document currently being worked on by the user and to open a new document, respectively. These examples are intended to illustrate how a user may utilize voice inputs to provide textual content and/or commands to an application <NUM> but do not limit the techniques disclosed herein to these specific examples. Additional use cases are illustrated in the examples that follow.

The NLPU <NUM> may be configured to use one or more speech models for converting an audio signal received by the application <NUM> into textual content representing the spoken content included in the audio signal. In the context of this application, the audio input being "received by the application" refers to an audio input being captured by one or more microphones associated with the computing device on which the application <NUM> is being executed. The audio signals may be provided as input to the application <NUM>, which may in turn provide all or a portion of the audio input to the CLPU <NUM>.

The NLPU <NUM> may process the audio content with more than model and compare the outputs of the models to determine the text to output by NLPU <NUM>. The NLPU <NUM> may be configured to include one or more local NLP models 180a that are implemented locally on the computing device on which the application <NUM> is being executed, one or more remote NLP models 180b provided by a remote server, or both. The one or more remote NLP models 180b may be implemented on one or more remote servers that are accessible via a network connection. In some implementations, the audio input that includes spoken content may be streamed from the user's computing device to a cloud-based service for speech recognition, such as the speech processing cloud services <NUM> illustrated in <FIG> and described in detail in the examples that follow. Text output and/or other information from the cloud-based service may be transmitted back to the computing device via the network connection. The one or more local NLP models 180a may be built into the application <NUM> and/or an operating system of the computing device on which the application <NUM> is being executed. In some implementations, a user may configure the NLPU <NUM> to utilize one or more custom NLP models provided by a user of the computing device.

Each NLP model may associate a confidence score with the textual output generated by the model. The NLPU <NUM> may select an output from among the outputs generated by the models used by the NLPU <NUM> as the text output of the NLPU <NUM>. This approach allows the CLPU <NUM> to decouple speech recognition from natural language understanding. The CLPU <NUM> can use various speech recognition engines to convert speech to text, and the resulting textual output can be provided to the machine learning language processing unit <NUM> for natural language understanding.

The MLLPU <NUM> is configured to receive the text output by the NLPU <NUM> and other contextual information associated with the application <NUM> to determine a contextual meaning for the text output by the NLPU <NUM>. The natural language processing techniques employed by the NLPU <NUM> may provide extremely accurate speech to text services, but without an understanding of the contextual meaning of the text the application <NUM> may incorrectly interpret the textual output of the NLPU <NUM>. For example, the textual output of the NLPU <NUM> may include textual content for the application <NUM>, commands to be executed by the application <NUM>, or a combination thereof. Without a contextual understanding of the textual output of the NLPU <NUM>, textual content may be incorrectly interpreted as a command or vice versa. Furthermore, even if a command is recognized as a command, the intent of the user with regard to the command may be unclear. Suppose, for example, that the user recites the sentence "machine learning is interesting" followed by the command "bold that. " The intent of the user is not clear based on the text alone. The user may intend to have the entire sentence rendered in bold font or the last word "interesting" rendered in bold font. Alternatively, the phrase "bold that" may not have been a command at all but instead may be the start of a new sentence of textual content. The NLPU <NUM> does not need to understand the underlying meaning or context of the text, the NPLU <NUM> just needs to provide an accurate transcription of the spoken input into text that may be analyzed by the MLPPU <NUM> for context.

The MLLPU <NUM> is configured to analyze the text generated by the NLPU <NUM> using one or more machine learning models, such as the one or more local ML models 190a and one or more remote ML models 190b, to provide a contextual understanding of how the words are being used in the text output by the NPLU <NUM>. The examples that follow illustrate the how the MLLPU <NUM> may be used to provide contextual understanding of the textual content, including interpretation of ambiguous command language included therein.

The MLLPU <NUM> may be configured to provide a context-sensitive understanding of commands included in the textual input. Suppose the user utters the phrase "bold that" as a command. What the word "that" refers to is ambiguous without additional context. The word "that" may be referring to rendering a last word or last phrase entered as bold or may be referring to a selected section of textual content. The MLLPU <NUM> is configured to process the text and contextual information provided by the application <NUM> to determine a meaning for "that" as used in the command. The contextual information provided by the application <NUM> includes usage information indicative of how the user was interacting with the application <NUM> when the command "bold that" was spoken. The contextual information may include an indication that user has selected a section of text content in the application. The MLLPU <NUM> may determine that the selected content was intended to be rendered in bold text in this context. The contextual information may also include a position of a cursor or insertion point (IP) in the document being edited at the time that the command was spoken. The context information includes an indication of the type of application being used by the user, and what type of content was being edited by the user at the time that the command was spoken. For example, if the IP indicates was positioned in a header of the document at the time that the phrase "bold that" was spoken, the MLLPU <NUM> may interpret this command to indicate that the entire header should be bolded based on a machine learning model used by the MLLPU <NUM> having been trained to understand that headers of the particular type of document being worked on by the user are typically emphasized using bold and/or underlined text. Other contextual information may be provided by the application <NUM> which may be used to disambiguate the meaning of the input text.

The machine learning models used by the MLLPU <NUM> can also incorporate culture-specific and domain-specific understanding of the textual inputs. For example, different models may be used for different geographical regions and/or languages which may have different sentence structure and grammatical rules. Region or culture specific may models may be used for different English or Spanish speaking regions of the world. For example, different models may be used for British English versus North American English. The models may be even more fine grained and support various dialects of English, that may include vocabulary, grammatical rules, and/or slang that is specific to those dialects, such as but not limited to American English, Canadian English, Scottish English, Irish English, and/or other regional dialects of the English language. Similarly, different models may be provided for Peninsular Spanish (European Spanish) versus New World Spanish dialects. Again, fine-grained models may support various dialects of Spanish that may include vocabulary, grammatical rules, and/or slang that is specific to those dialects, such as but not limited to Chilean Spanish, Mexican Spanish, and Andalusian Spanish. Other models may be configured to support other languages and/or dialects thereof. The context information provided by the application <NUM> may include region information indicative of a geographical location in which the computing device is located, a language selected by the user, and/or other information that can be used to determine a model that may be used to determine a context of the input from the NLPU <NUM>.

The MLLPU <NUM> is/are configured to provide a context-sensitive understanding of textual inputs to determine whether a part of an audio input is intended to be textual content or intended to be a command acted upon by the application <NUM> in a context-sensitive manner. For example, the text output by the NLPU <NUM> may include "This is great! Italicize the last word. " Ideally, the first half of the text, which includes the text "This is great!", should be interpreted as text content, while the second half of the text input should be interpreted as a command, which includes the text "Italicize the last word. " The machine learning models used by the MLLPU <NUM> can be trained to identify a set of commands words or phrases that may typically be spoken by a user of the application <NUM>. The machine learning models may be trained to identify a different set of commands for different types of applications, such as but not limited a web browser versus a messaging application. The machine learning models may be trained to identify specific sets of commands for specific applications.

The MLLPU <NUM> may be configured to provide a context-sensitive understanding of textual inputs to disambiguate phrases that sound similar and may be misinterpreted by the NPLU <NUM>. For example, the command "insert table" sounds very much like the word "insertable" which may be included in textual input spoken by the user. The natural language models used by the NPLU <NUM> may or may not accurately determine whether utterance was meant to be a command or was meant to be a word included in the text. However, the machine learning models used by the MLLPU <NUM> may be trained to identify ambiguous words or phrases in included in the text output by the NLPU <NUM> and may make a determination whether the user intended to issue the command "insert table" or merely to include the word "insertable" in the textual content provided to the application <NUM>. The MLLPU <NUM> may use the context information provided by the application <NUM> in addition to the text provided by the NPLU <NUM> to make a determination what the user intended to say and whether that utterance was a command or textual content. The machine learning model(s) used by the MLLPU <NUM> may look at the text before and/or after the utterance when determining the context. For example, the user may use the word "insertable" in a sentence with certain words, such as "insertable into" which may indicate that user intended to use the word "insertable" rather than issue the command "insert table. " The machine learning model(s) may take into account the type of application being used by the user when disambiguating between multiple possible utterances. For example, the model may determine that the command "insert table" was more probable where the user is working in a spreadsheet application and is less likely if the user is working in a messaging application. These examples are intended to illustrate on example in which the machine learning model(s) of the MLLPU <NUM> may be used to resolve ambiguities in the language uttered by the user and do not limit the models to these specific examples.

The MLLPU <NUM> may be configured to trigger a disambiguation process in some implementations. The MLLPU <NUM> may be configured to trigger a disambiguation process in response to the model(s) used by the MLLPU <NUM> outputting multiple possible outputs where the MLLPU <NUM> is unable to distinguish between the outputs. The model(s) may have assigned similar confidence score or none of the outputs may have been assigned a high enough confidence score to make a decision. The MLLPU <NUM> or the CLPU <NUM> may be configured to render a user interface on the computing device of the user to present a list of choices to the user from which the user may select what was actually said by the user. The user interface may also include an option where the user may select enter text indicating what was actually said by the user if the list of options presented by the MLLPU <NUM> or the CLPU <NUM> is incorrect. In some implementations, if the user utterance is determined to be a command, the MLLPU <NUM> or the CLPU <NUM> may provide suggestions that more clearly articulate the command user was trying to say once the input has been disambiguated.

The MLLPU <NUM> may be configured to identify unimplemented commands based on user utterances and observed multi-modal actions. For example, the MLLPU <NUM> may obtain keyboard and/or mouse interactions with the application <NUM> with the context information received from the application <NUM>. The MLLPU <NUM> may analyze the user utterances and interactions with the user interface of the application <NUM> to identify additional commands that may not yet be supported by the MLLPU <NUM> but the model(s) used by the MLLPU <NUM> could be trained to support voice commands for such utterances. This information may be crowdsourced across users by reporting such unsupported utterances to a cloud-based service associated with the MLLPU <NUM>. For example, if more than a threshold number of users utter the command "watermark" and then perform a series of mouse and/or keyboard commands to execute the watermark functionality when the user realizes that the voice command is not supported, then the model(s) used by the MLLPU <NUM> may be trained to recognize "watermark" as a command rather than just textual content. In implementations where the MLLPU <NUM> utilized one or more models stored locally on the user device, the NLPU <NUM> may receive updates to the local model(s) from the cloud-based service to update the model(s) used by the MLLPU <NUM> to support commands added through crowdsourcing.

<FIG> is a block diagram illustrating additional details of the example computing environment illustrated in <FIG>. The example illustrated in <FIG> illustrates the data elements that are passed among the various components of the computing environment <NUM>.

The application <NUM> captures an audio input <NUM> from a user. The audio input <NUM> may be captured using a microphone of the user's computing device. The audio input <NUM> includes spoken content that includes a command, textual content, or both. The application <NUM> may provide the audio input <NUM> and context information <NUM> to the CLPU <NUM>. The context information <NUM> may include additional information identifying the application <NUM>, regional information and/or language information identifying a language and/or dialect that may be spoken by the user, and/or other information that may be provided by the application that may be used to disambiguate the meaning of the spoken content. The application <NUM> may obtain at least a portion of the context information from an operating system of the computing device. The context information <NUM> may include cursor or insertion point (IP) location information and/or other information that indicates how the user was interacting with the application prior to, during, or after issuing a voice command. The cursor or IP location information may be used to disambiguate how the command is to be applied to textual content being dictated and/or revised by the user.

The CLPU <NUM> receives the audio input <NUM> and the optional context information <NUM> from the application <NUM>. The CLPU <NUM> may output audio input(s) <NUM> to the NLPU <NUM>. The NLPU <NUM> is configured to convert the spoken language included in the audio input(s) <NUM> to textual content. The audio input(s) <NUM> may be the same as the audio input <NUM> or the CLPU <NUM> may process the audio input <NUM> to generate the audio input(s) <NUM> for one or more natural language processing modules associated with the NLPU <NUM>. For example, the CLPU <NUM> may preprocess the audio input <NUM> to filter out background noise, to convert the audio input <NUM> to a format that may be processed by one or more natural language processing models utilized by the NPLU <NUM>, and/or other processing of the audio signal that may facilitate the one or more natural language processing models recognition of spoken content in the audio input <NUM>. The CLPU <NUM> may output more than one audio input <NUM> for the NPLU <NUM> for use by one or more natural language processing models.

The NPLU <NUM> may be associated with one or more local natural language processing (NLP) models 180a, one or more remote NLP models 180b, or both. The local NLP models 180a may be implemented on the user's computing device, such as the user device <NUM> depicted in <FIG>. The remote NLP models 180b may be implemented on one or more servers remote from the user's computing device that are accessible via one or more network connections. The remote NLP models 180b may be implemented by the speech processing cloud services <NUM> illustrated in <FIG> and discussed in detail in the examples the follow.

The NPLU <NUM> may be configured to provide the audio input(s) 150a to the local NLP models 180a and the audio input(s) 150b to the remote NLP models 180b. In some implementations, the NPLU <NUM> may preprocess the audio input(s) received from the CLPU <NUM> to filter out background noise, to convert the audio input(s) <NUM> to a format that may be processed by one or more local or remote natural language processing models utilized by the NPLU <NUM>, and/or other processing of the audio signal that may facilitate the one or more natural language processing models recognition of spoken content in the audio input <NUM>. Each NLP model used by the NLPU <NUM> may provide a textual output and a confidence score to the NPLU <NUM>, such as the textual output(s) and confidence score(s) 155a from the local NLP models 180a and the textual output(s) and confidence score(s) 155b from the remote NLP models 180b. The textual output from each NLP model represents that particular model's interpretation of the spoken content included in the audio signal provided to the model. The NLP model may also output a confidence score that represents an estimated probability of correctness of the textual output by that model. The calculation of the confidence score is dependent upon the implementation of the NLP model. The NPLU <NUM> may be configured to select a textual output from one of the NLP models, where more than one NL model is used, to output as the text output <NUM>. The NPLU <NUM> may select the textual output associated with the highest confidence score. In some implementations, the NPLU <NUM> may only use one remote or one local NLP model to generate the textual output. In such implementations, the NLPU <NUM> may output the textual output from that model without considering a confidence score associated with the textual output.

The CLPU <NUM> may provide the text output <NUM> as a text input <NUM> to the MLLPU <NUM>. The MLLPU <NUM> provides the text input <NUM> to one or more local machine learning (ML) models 190a and/or one or more remote ML models 190b. The one or more local ML models 190a may be implemented on the user's computing device, such as the user device <NUM> depicted in <FIG>. The remote ML models 190b may be implemented on one or more servers remote from the user's computing device that are accessible via one or more network connections. The remote ML models 190b may be implemented by the speech processing cloud services <NUM> illustrated in <FIG> and discussed in detail in the examples the follow.

Each ML model is configured to receive the text input <NUM> and receives the context information <NUM> including application usage information from the application <NUM>. As discussed above, the context information <NUM> may include language and/or dialect information, location of the cursor or IP within the text of a document being edited using the application, and/or other information that may be used to determine the context of the textual content output by the NLPU <NUM>. The context information may also include at least a portion of the textual content that has already included in the document being edited in the application <NUM>. The textual content included with the context information may be used by the ML model(s) in situations where the spoken content included in the audio input includes a command but no text to which the command is to be applied. Each ML model may output context information and a confidence score associated with that confidence information.

The local ML models 190a may provide the context information and confidence score(s) 170a, and the remote ML models 190b may provide the context information and confidence score(s) 170b. The context information may include an indication of whether the textual input provided to the ML model included at least one command and information indicating how the user intended to apply that command to textual content included in the textual input and/or existing textual content that has already been added to the document being edited in the application <NUM>. The confidence score represents an estimated probability of correctness of the context information output by that model.

The MLLPU <NUM> may select context information received from a plurality of ML models based on the confidence scores associated with the ML models. The MLLPU <NUM> may select context information that is associated with a highest confidence score and output that context information as context information <NUM> to the CLPU <NUM>. The CLPU <NUM> may use the context information <NUM> to determine text content and/or commands <NUM> to be provided to the application <NUM>. The text content includes textual content that was dictated by the user and includes in the audio input <NUM>. The command information may include one or more commands to be performed on textual content by the application <NUM>. The command information may identify which text on which each command is to be performed.

<FIG> illustrates a block diagram of an example computing environment <NUM> in which the techniques disclosed herein may be implemented. The application <NUM> may be implemented on the user device <NUM> or the application <NUM> may be implemented as a cloud-based application <NUM> that is accessible from a web browser or other similar interface on the user device <NUM>. As discussed in the preceding examples, the user device <NUM> may be a personal computer (PC), a tablet computer, a laptop computer, a netbook, a gaming and/or entertainment system, a smart phone, a mobile device, a wearable device, or other type of computing device. The speech processing cloud services <NUM> may be configured to implement at least a portion of the CLPU <NUM> illustrated in <FIG> and <FIG>. The CLPU <NUM> may be implemented on the user device <NUM> and/or on the speech processing cloud services <NUM>. Furthermore, one or more models used by the NLPU <NUM> and/or the MLLPU <NUM> may be implemented by the speech processing cloud service <NUM> and/or on the user device <NUM>.

<FIG> is a flow diagram of an example process <NUM> that may be implemented by the CLPU <NUM>. In the example illustrated in <FIG>, the command "bold" or "bold that" is detected in the audio input <NUM> received from the application <NUM>. The process <NUM> starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, the user dictates textual content <NUM>, and then pauses at operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

The user may then express an intent to perform a command in operation <NUM>. The command may be uttered in a second utterance after the user pauses at operation <NUM>. While the example illustrated in <FIG> includes a pause at operation <NUM>, the techniques disclosed herein do not require a pause between the textual content dictated by the user in operation <NUM> and utterance of the command in operation <NUM>. As discussed in the preceding examples, the audio input that includes the spoken content to be processed includes both textual content and one or more commands rather than separate inputs separated by a pause.

In operation <NUM>, the user expresses the intent "do <ActionX>" by issuing a command instructing the application <NUM> to perform a specified action. In this example, the user issues the command "Bold" or "Bold that" indicating that the user would like to render text in bold font. The process <NUM> illustrates an example process that the CLPU <NUM> may perform to determine which textual content is being referred to by the user to be rendered in bold font. The CLPU <NUM> and/or the MLLPU <NUM> may determine that the second utterance is a command based on the pause in the user speech. The MLLPU <NUM> can use this information to infer that the user has expressed an intent to perform a command at operation <NUM>. The process may continue with operation <NUM>.

The operation illustrated in <FIG> may alternatively flow from the original state <NUM>, to one of several operations in which the user sets the position of the cursor / IP before uttering a command. In a first example, the user places the cursor / IP within a word (operation <NUM>) and then issues the voice command "bold that" (operation <NUM>). In a second example, the user places the cursor / IP after a word (operation <NUM>) and then issues the voice command "bold that" (operation <NUM>). In a third example, the user places the cursor / IP after a word (operation <NUM>) and issues the command "bold") (operation <NUM>). The process may continue with operation <NUM> from operations <NUM>, <NUM>, or <NUM>.

The process may continue with operation <NUM>, in which a determination is made whether the intent has already been executed on the text. The command may be ignored at operation <NUM> if the intent has already been executed on the textual content. The MLLPU <NUM> and/or the CLPU <NUM> may then optionally provide user education or assistance based on the intent of the user inferred by the MLLPU <NUM> based on the textual input provided by the user and the contextual information received from the application <NUM>. For example, the MLLPU <NUM> may display and/or use text-to-speech to provide audio feedback to the user providing alternative wording to more clearly express the command language and/or to provide guidance how to express another alternative command that the user may have intended to issue.

If the command was not already executed on the text, the process may continue with one of operations <NUM>, <NUM>, <NUM>, or <NUM> depending upon the preceding operations. If operation <NUM> preceded operation <NUM>, then the operation <NUM> may be executed, and the text of the word in which the cursor / IP was located is rendered in bold font. If the operation <NUM> preceded operation <NUM>, then the operation <NUM> may be executed in which text of the word preceding the location of the cursor / IP was located is rendered in bold font. If the operation <NUM> preceded the operation <NUM>, then the operation <NUM> may be executed and "bold" font style may be activated so that subsequent text which is added via dictation or by typing will be rendered in bold font. If the operation <NUM> preceded the operation <NUM>, then the operation <NUM> may be executed and the last partial sentence dictated and/or typed by the user may be rendered in bold font.

The process may continue with the operation <NUM> in which the MLLPU <NUM> may confirm the behavior of the model used was correct to reinforce the behavior of the model. Reinforcing the behavior of the model may help improve the performance of the model. The specific means for reinforcing the behavior of the model depends upon the type of model(s) being used by the MLLPU <NUM>. The operation <NUM> may be followed by operation <NUM> in which the following operations may be performed: (<NUM>) if the text was selected, keep the text selected after rendering in bold font, and (<NUM>) if the cursor / IP was within the text when the "bold" command was uttered by the user, then keep the cursor / IP at that same location.

<FIG> is a flow diagram of an example process <NUM> that may be implemented by the CLPU <NUM>. In the example illustrated in <FIG>, the command "select that" or "select last sentence" or "select next sentence" is detected in the audio input received from the application <NUM>. The process <NUM> starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, user dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

The process <NUM> splits into two flows after operation <NUM>: the user expresses "select this" or "select that" at operation <NUM> or the user expresses "select next sentence" or "select last sentence" at operation <NUM>. As discussed in the preceding examples, operation <NUM> is optional. The techniques disclosed herein do not require a pause between the textual content dictated by the user in operation <NUM> and utterance of the command in operations <NUM> or <NUM>. As discussed in the preceding examples, the audio input that includes the spoken content to be processed may include both textual content and one or more commands.

The process may then split into multiple flows following operation <NUM> based on the position of the cursor / IP at the time that command was issued by the user. In operation <NUM>, the cursor / IP is determined to be in the middle of a sentence. In operation <NUM>, the cursor / IP is determined to be located in between sentences and after final punctuation. Final punctuation is indicative of the end of a sentence. The final punctuation may be language specific. For example, a period is indicative of the end of the sentence in English, Spanish, and other languages, while a kuten or maru is used to indicate the end of a sentence in Japanese. Other types of indicators may also be used, such as a paragraph indicator in the document, which indicates that a new paragraph has begun in the text. In both of the preceding scenarios, the sentence before or following the cursor / IP or in which the cursor / IP is located will be selected dependent upon whether the user indicated that the "last" or the "next" sentence be selected in operation <NUM>. In operation <NUM>, the cursor / IP is determined to be located at the end of a document when the command is issued. In this scenario, any text located between the current location of the cursor / IP and the previous end punctuation is selected in operation <NUM>. The process may continue with the operation <NUM> in which the MLLPU <NUM> may confirm the behavior of the model used was correct to reinforce the behavior of the model. As discussed in the preceding examples, reinforcing the behavior of the model may help improve the performance of the model. The specific means for reinforcing the behavior of the model depends upon the type of model(s) being used by the MLLPU <NUM>.

The process may also split into multiple flows following operation <NUM> based on the position of the cursor / IP at the time that command was issued by the user. In operation <NUM>, the cursor / IP is determined to be in the middle of a sentence. The sentence in which the cursor / IP is located will be selected in operation <NUM>. In operation <NUM>, the cursor / IP is determined to be located in between sentences and after final punctuation, and in operation <NUM>, the cursor / IP is determined to be located at the end of a document when the command is issued. In either of these two scenarios, any text located between the current location of the cursor / IP and the previous end punctuation is selected in operation <NUM>. The process may continue with the operation <NUM> in which the MLLPU <NUM> may confirm the behavior of the model used was correct to reinforce the behavior of the model.

<FIG> is a flow diagram of an example process <NUM> that may be implemented by the CLPU <NUM>. In the example illustrated in <FIG>, the command "select the last sentence and bold" is detected in the audio input received from the application <NUM>. The process starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, user dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

The user may then express an intent to perform a command in operation <NUM>. The command may be uttered in a second utterance after the user pauses at operation <NUM>. As discussed in the preceding examples, operation <NUM> is optional. The techniques disclosed herein do not require a pause between the textual content dictated by the user in operation <NUM> and utterance of the command in operation <NUM>. As discussed in the preceding examples, the audio input that includes the spoken content to be processed may include both textual content and one or more commands.

In operation <NUM>, the user expresses the intent to "Select <Y> and do <Action X>. " In this example, the value of "<Y>" is the last sentence entered and the value of "<Action X>" is the "bold" command which renders the selected text in a bold font. The MLLPU <NUM> may determine that the second utterance is a command based on the pause in the user speech, the use of the keyword "select" which the model may be trained to identify as a command. The MLLPU <NUM> can use this information to infer that the user has expressed an intent to perform a command.

A list of fixed and/or machine learning driven utterances may then be loaded by the MLLPU <NUM> at operation <NUM>. The MLLPU <NUM> may use a list of fixed utterance for a particular application or domain. The MLLPU <NUM> may also access a list of ML driven utterances that were identified by the machine learning model(s) but were not included in the fixed list, such as in the "watermark" example discussed above.

A determination may then be made whether the intent has already been executed on the textual content at operation <NUM>. The command may be ignored at operation <NUM> if the intent has already been executed on the textual content. The MLLPU <NUM> and/or the CLPU <NUM> may then optionally provide user education / assistance based on the intent of the user inferred by the MLLPU <NUM> based on the textual input provided by the user and the contextual information received from the application <NUM>. Based on the user utterances and the contextual information, the MLLPU <NUM> may infer that the user was actually intending to issue a command "<Y>" instead of the command that was issued. In this example, the user may have meant to issue the command "select the last sentence" rather than "select the last sentence and bold. " Furthermore, the MLLPU <NUM> inferred that the "last sentence" refers to the previously dictated sentence rather than a last sentence in the document based on the content information provided by the application <NUM> and the position of the insertion point / cursor at the time that the command was entered.

If the intent has not already been executed on the textual content at operation <NUM>, then the intended operation can be performed at operation <NUM>. As indicated above, the previously dictated sentence is rendered in bold font. The MLLPU <NUM> may confirm the behavior of the model used was correct to reinforce the behavior of the model in operation <NUM>.

The MLLPU <NUM> may also keep the cursor / IP at a location where the cursor / IP was located prior to receipt of the command in operation <NUM>. The CLPU <NUM> may monitor the user interactions with the application <NUM> at operation <NUM> to determine whether the user reverts the changes that were made in operation <NUM>, which may indicate that the MLLPU <NUM> misinterpreted the user intent or that the user misspoke. The user may revert the changes through a set of keystrokes (e.g. pressing the control key + Z) and/or through a series of mouse clicks (e.g. selecting an undo menu item). If the user does not revert the changes, the MLLPU <NUM> may confirm the behavior of the model used was correct to reinforce the behavior of the model. If the user does revert the changes and the user repeats the utterance made before the undo, then in operation <NUM> the utterance may be inferred to be textual content rather than a command and the utterance is rendered as textual content on the user interface of the application <NUM>. If the user does revert the changes, the MLLPU <NUM> may provide a textual and/or audible suggestion that the user might also want to say an alternative command suggested by the MLLPU <NUM>.

<FIG> illustrates another example process <NUM> that is similar to the example process <NUM> of <FIG>. The process illustrated in <FIG> illustrates an example in which the command "bold the last five words" is detected in the audio input received from the application <NUM>. The process starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, user dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

In operation <NUM>, the user expresses the intent to "Select <Y> and <Action X>. " In this example, the value of "<Y>" is the last five words and the value of "<Action X>" is the "bold" command which renders the selected text in a bold font. The MLLPU <NUM> may determine that the second utterance is a command based on the pause in the user speech, the use of the keyword "bold" which the model may be trained to identify as a command. The MLLPU <NUM> can use this information to infer that the user has expressed an intent to perform a command.

The example illustrated in <FIG> proceeds similarly to the example illustrated in <FIG> from this point. Step <NUM>-<NUM> are similar to steps <NUM>-<NUM> illustrated in the example of <FIG>. In operation <NUM>, the last five words dictated may be rendered in bold font.

<FIG> illustrates another example process <NUM> in which a "capitalize" command is detected in the audio input received from the application <NUM>. The process starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, user dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

In operation <NUM>, the user expresses the intent to "Select <text> and <Intent> Capitalize this. " In this example, the value of "<text>" may refer to a subset of the text content that has been dictated, such as a last sentence, this word, or these words, and the use of the "Capitalize" command indicates that the selected text is to be capitalized. The wording of the command may vary depending upon what is being selected. For example, the command may be "capitalize this word," "capitalize this sentence," etc..

In operation <NUM>, a determination can be made whether the user selected less than an entire sentence. If the user did select less than the entire sentence, then in operation <NUM> logic to capitalize each of the selected words may be invoked. Otherwise, sentence capitalization logic may be invoked to provide proper capitalization based on sentence structure in operation <NUM>. In operation <NUM>, the text may be kept selected after performing either operation <NUM> or <NUM> to capitalize the text. The CLPU <NUM> may monitor the user interactions with the application <NUM> at operation <NUM>, similar to operation <NUM> of the process illustrated in <FIG>, to determine whether the user reverts the changes that were made in operations <NUM> or <NUM>. If the user does revert the changes and the user repeats the utterance made before the undo, then in operation <NUM> the utterance may be inferred to be textual content rather than a command and the utterance is rendered as textual content on the user interface of the application <NUM>.

<FIG> illustrates another example process <NUM> in which a "delete" command is detected in the audio input received from the application <NUM>. The process starts with the application <NUM> being in an original state <NUM>. The original state may include content, textual or otherwise, that has already added to a document by a current user or another user. In this example, the flow splits depending upon the actions taken by the user.

User dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and make a determination that the pause is an indication that the user has completed the dictation of the textual content.

As discussed in the preceding examples, operation <NUM> is optional. The techniques disclosed herein do not require a pause between the textual content dictated by the user in operation <NUM> and utterance of the command in operations <NUM> or <NUM>. As discussed in the preceding examples, the audio input that includes the spoken content to be processed may include both textual content and one or more commands.

The CLPU <NUM> then detects that the user utters the command "delete" in operation <NUM> or "delete that" in operation <NUM>. The CLPU <NUM> determines that the user intended to delete the entire previous utterance in response to operation <NUM> and provides an indication to the application <NUM> to delete the word in operation <NUM>. The CLPU <NUM> determines that the user intended to delete the word before the location of the cursor / IP is located in response to operation <NUM> and provides an indication to the application <NUM> to delete the word in operation <NUM>.

In operation <NUM>, a determination is made that the user has placed the cursor / IP after a word in the textual content of the document. The CLPU <NUM> then detects that the user utters the command "delete" or "delete that" in operation <NUM>. The CLPU <NUM> determines that the user intended to delete the word before the location of the cursor / IP is located and provides an indication to the application <NUM> to delete the word in operation <NUM>.

In operation <NUM>, a determination is made that the user has placed the cursor / IP within a word in the textual content of the document. The CLPU <NUM> then detects that the user utters the command "delete" or "delete that" in operation <NUM>. The CLPU <NUM> determines that the user intended to delete the word in which the cursor / IP is located and provides an indication to the application <NUM> to delete the word in operation <NUM>.

<FIG> illustrates another example process <NUM> in which a "select" command is detected in the audio input received from the application <NUM> indicating that the user would like to select a word or a phrase from the text content already dictated or typed into the application <NUM>. The process starts with the application <NUM> being in an original state in operation <NUM>. In this example, user dictation of textual content is detected in operation <NUM>, and a pause in the user dictation is detected in operation <NUM>. The CLPU <NUM> may determine, based on the output of the MLLPU <NUM>, that the textual content dictated by the user comprises textual content to be rendered by the application <NUM>. The MLLPU <NUM> may detect the pause in the audio content at operation <NUM> and may make a determination that the pause is an indication that the user has completed the dictation of the textual content.

The flow of the process <NUM> may split after operation <NUM> in which a pause in the user dictation is detected depending upon the user command detected, and operation <NUM> or <NUM> may be performed. As discussed in the preceding examples, operation <NUM> is optional. The techniques disclosed herein do not require a pause between the textual content dictated by the user in operation <NUM> and utterance of the command in operations <NUM> or <NUM>. As discussed in the preceding examples, the audio input that includes the spoken content to be processed may include both textual content and one or more commands.

In operation <NUM>, the command "select <text>" is detected in operation <NUM>, where "<text>" represents word or phrase in the text to be selected. The CLPU <NUM> determines that the user intended to select the closest instance of the <text> that precedes the current location of the cursor / IP, and the detected instance of the <text> in operation <NUM>.

In operation <NUM>, the command "select <text1> to <text2>" is detected in operation <NUM>, where "<text1>" represents a first word or phrase in the text to be selected and "<text2>" represents and ending word or phrase to be selected. The CLPU <NUM> the closest instance of the <text2> that precedes the current location of the cursor / IP in operation <NUM>. The CLPU <NUM> determines that closest instance of the <text1> that precedes the location of <text2> in operation <NUM>. The CLPU <NUM> then sends an indication to the application <NUM> that the text from <text1> to <text2> should be selected in operation <NUM>.

<FIG> illustrates another example of voice-command chaining that may be implemented by the CLPU <NUM> of <FIG> and <FIG>. The CLPU <NUM> can be configured to take into account the context in which a voice command issued by the user to determine how formatting updates are to be applied to the target text. <FIG> illustrates an example of a user issuing the voice command "remove underline and bold. " This command could be interpreted in several different ways. One interpretation is that the remove command applies to both underline and bold. Another interpretation is that the remove command applies only to the underline and that bold is separate command to render the text in bold font.

As discussed in the preceding examples, the application can provide provides contextual information to the CLPU <NUM> which is provided to the MLLPU <NUM> for disambiguating what the user said based on the context of how the command is used and how the user is interacting with the application. The context information can include current cursor / IP location information, previously issued commands, current textual context being rendered by the application, and/or other information that may be indicative of the user's intent. The context can also be determined based on how the text that is the subject of the command is already formatted.

The command "remove text and bold" when applied to text that is both underlined and formatted as in the first example illustrated in <FIG>, may understood based on the context to remove both the bold and the underlying from the text. Furthermore, the same command may be interpreted different when applied to text that is underlined but not bold as in the second example illustrated in <FIG>. In the second example, the CLPU <NUM> and/or the MLLPU <NUM> may recognize that the text is underlined only and treat the world "bold" as a command. Thus, the underlining is removed from the text and the text is then rendered as bold text. This example is intended to illustrate how the existing formatting of the textual content of the application <NUM> may be used to provide context as to how a command should be interpreted and the techniques disclosed herein are not limited to this specific example. Other types of formatting related commands may be chained together in an audio input provided by the user and the existing formatting of the text can be used to disambiguate the commands.

<FIG> is a flow chart illustrating an implementation of an example process <NUM> executed by a data processing system for processing voice content received from a user. The process <NUM> may be implemented by the CLPU <NUM> illustrated in the preceding examples. The process <NUM> may be implemented by a data processing system similar to the data processing system <NUM> of <FIG>.

The process <NUM> includes an operation <NUM> of receiving a first audio input from a user comprising spoken content. As discussed in the preceding examples, a user may dictate textual content for an application and/or may provide voice commands to be executed by the application. Thus, the spoken content may include a command, textual content, or both.

The process <NUM> includes an operation <NUM> of analyzing the first audio input using one or more natural language processing models to produce a first textual output comprising a textual representation of the first audio input. The first audio input may be analyzed by one or more natural language processing models of the NPLU <NUM> to determine a first textual output that represents the textual input and/or commands spoken by the user.

The process <NUM> includes an operation <NUM> of analyzing the first textual output using one or more machine learning models to determine first context information of the first textual output. The first context information provides an indication of whether the first textual output includes the command and an indication of how the user intended to apply the command to content in an application as determined by the one or more machine learning models of the MLLPU <NUM> that analyzed the textual output of the NPLU <NUM>. The contextual information may also include one or more parameters associated with a command included in the first textual output, which can be used by the application <NUM> to determine how the command is to be applied to the textual content included in the first textual output and/or already received by the application <NUM>. For example, the context information may include in indication of a section of the textual content to which the command is to be applied.

The process <NUM> includes an operation <NUM> of processing the first textual output in the application based on the first context information. The application <NUM> receives the first textual input and the first context information from the CLPU <NUM>. The application <NUM> uses the first context information to determine whether the first textual output includes textual content to be rendered by the application <NUM> and/or one or more commands to be executed by the application <NUM>. The one or more commands may include one or more parameters for the commands. The one or more commands may also be executed on textual content that is included in the first textual output and/or on other textual content that has already been received by the application <NUM> from the same or another user.

The detailed examples of systems, devices, and techniques described in connection with <FIG> are presented herein for illustration of the disclosure and its benefits. Such examples of use should not be construed to be limitations on the logical process embodiments of the disclosure, nor should variations of user interface methods from those described herein be considered outside the scope of the present disclosure. It is understood that references to displaying or presenting an item (such as, but not limited to, presenting an image on a display device, presenting audio via one or more loudspeakers, and/or vibrating a device) include issuing instructions, commands, and/or signals causing, or reasonably expected to cause, a device or system to display or present the item. In some embodiments, various features described in <FIG> are implemented in respective modules, which may also be referred to as, and/or include, logic, components, units, and/or mechanisms. Modules may constitute either software modules (for example, code embodied on a machine-readable medium) or hardware modules.

In some examples, a hardware module may be implemented mechanically, electronically, or with any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is configured to perform certain operations. For example, a hardware module may include a special-purpose processor, such as a field-programmable gate array (FPGA) or an Application Specific Integrated Circuit (ASIC). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations and may include a portion of machine-readable medium data and/or instructions for such configuration. For example, a hardware module may include software encompassed within a programmable processor configured to execute a set of software instructions. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (for example, configured by software) may be driven by cost, time, support, and engineering considerations.

Accordingly, the phrase "hardware module" should be understood to encompass a tangible entity capable of performing certain operations and may be configured or arranged in a certain physical manner, be that an entity that is physically constructed, permanently configured (for example, hardwired), and/or temporarily configured (for example, programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, "hardware-implemented module" refers to a hardware module. Considering examples in which hardware modules are temporarily configured (for example, programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where a hardware module includes a programmable processor configured by software to become a special-purpose processor, the programmable processor may be configured as respectively different special-purpose processors (for example, including different hardware modules) at different times. Software may accordingly configure a processor or processors, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time. A hardware module implemented using one or more processors may be referred to as being "processor implemented" or "computer implemented.

Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (for example, over appropriate circuits and buses) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory devices to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output in a memory device, and another hardware module may then access the memory device to retrieve and process the stored output.

In some examples, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. For example, at least some of the operations may be performed by, and/or among, multiple computers (as examples of machines including processors), with these operations being accessible via a network (for example, the Internet) and/or via one or more software interfaces (for example, an application program interface (API)). The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across several machines. Processors or processor-implemented modules may be in a single geographic location (for example, within a home or office environment, or a server farm), or may be distributed across multiple geographic locations.

<FIG> is a block diagram <NUM> illustrating an example software architecture <NUM>, various portions of which may be used in conjunction with various hardware architectures herein described, which may implement any of the above-described features. <FIG> is a non-limiting example of a software architecture and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture <NUM> may execute on hardware such as a machine <NUM> of <FIG> that includes, among other things, processors <NUM>, memory <NUM>, and input/output (I/O) components <NUM>. A representative hardware layer <NUM> is illustrated and can represent, for example, the machine <NUM> of <FIG>. The representative hardware layer <NUM> includes a processing unit <NUM> and associated executable instructions <NUM>. The executable instructions <NUM> represent executable instructions of the software architecture <NUM>, including implementation of the methods, modules and so forth described herein. The hardware layer <NUM> also includes a memory/storage <NUM>, which also includes the executable instructions <NUM> and accompanying data. The hardware layer <NUM> may also include other hardware modules <NUM>. Instructions <NUM> held by processing unit <NUM> may be portions of instructions <NUM> held by the memory/storage <NUM>.

The frameworks <NUM> (also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications <NUM> and/or other software modules. For example, the frameworks <NUM> may provide various graphic user interface (GUI) functions, high-level resource management, or high-level location services. The frameworks <NUM> may provide a broad spectrum of other APIs for applications <NUM> and/or other software modules.

The applications <NUM> include built-in applications <NUM> and/or third-party applications <NUM>. Examples of built-in applications <NUM> may include, but are not limited to, a contacts application, a browser application, a location application, a media application, a messaging application, and/or a game application. Third-party applications <NUM> may include any applications developed by an entity other than the vendor of the particular platform. The applications <NUM> may use functions available via OS <NUM>, libraries <NUM>, frameworks <NUM>, and presentation layer <NUM> to create user interfaces to interact with users.

Some software architectures use virtual machines, as illustrated by a virtual machine <NUM>. The virtual machine <NUM> provides an execution environment where applications/modules can execute as if they were executing on a hardware machine (such as the machine <NUM> of <FIG>, for example). The virtual machine <NUM> may be hosted by a host OS (for example, OS <NUM>) or hypervisor, and may have a virtual machine monitor <NUM> which manages operation of the virtual machine <NUM> and interoperation with the host operating system. A software architecture, which may be different from software architecture <NUM> outside of the virtual machine, executes within the virtual machine <NUM> such as an OS <NUM>, libraries <NUM>, frameworks <NUM>, applications <NUM>, and/or a presentation layer <NUM>.

<FIG> is a block diagram illustrating components of an example machine <NUM> configured to read instructions from a machine-readable medium (for example, a machine-readable storage medium) and perform any of the features described herein. The example machine <NUM> is in a form of a computer system, within which instructions <NUM> (for example, in the form of software components) for causing the machine <NUM> to perform any of the features described herein may be executed. As such, the instructions <NUM> may be used to implement modules or components described herein. The instructions <NUM> cause unprogrammed and/or unconfigured machine <NUM> to operate as a particular machine configured to carry out the described features. The machine <NUM> may be configured to operate as a standalone device or may be coupled (for example, networked) to other machines. In a networked deployment, the machine <NUM> may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a node in a peer-to-peer or distributed network environment. Machine <NUM> may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and/or entertainment system, a smart phone, a mobile device, a wearable device (for example, a smart watch), and an Internet of Things (IoT) device. Further, although only a single machine <NUM> is illustrated, the term "machine" includes a collection of machines that individually or jointly execute the instructions <NUM>.

The machine <NUM> may include processors <NUM>, memory <NUM>, and I/O components <NUM>, which may be communicatively coupled via, for example, a bus <NUM>. The bus <NUM> may include multiple buses coupling various elements of machine <NUM> via various bus technologies and protocols. In an example, the processors <NUM> (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 1212a to 1212n that may execute the instructions <NUM> and process data. In some examples, one or more processors <NUM> may execute instructions provided or identified by one or more other processors <NUM>. The term "processor" includes a multicore processor including cores that may execute instructions contemporaneously. Although <FIG> shows multiple processors, the machine <NUM> may include a single processor with a single core, a single processor with multiple cores (for example, a multicore processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine <NUM> may include multiple processors distributed among multiple machines.

As used herein, "machine-readable medium" refers to a device able to temporarily or permanently store instructions and data that cause machine <NUM> to operate in a specific fashion, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, cache memory, network-accessible or cloud storage, other types of storage and/or any suitable combination thereof. The term "machine-readable medium" applies to a single medium, or combination of multiple media, used to store instructions (for example, instructions <NUM>) for execution by a machine <NUM> such that the instructions, when executed by one or more processors <NUM> of the machine <NUM>, cause the machine <NUM> to perform and one or more of the features described herein. Accordingly, a "machine-readable medium" may refer to a single storage device, as well as "cloud-based" storage systems or storage networks that include multiple storage apparatus or devices. The term "machine-readable medium" excludes signals per se.

The I/O components <NUM> may include a wide variety of hardware components adapted to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components <NUM> included in a particular machine will depend on the type and/or function of the machine. For example, mobile devices such as mobile phones may include a touch input device, whereas a headless server or IoT device may not include such a touch input device. The particular examples of I/O components illustrated in <FIG> are in no way limiting, and other types of components may be included in machine <NUM>. The grouping of I/O components <NUM> are merely for simplifying this discussion, and the grouping is in no way limiting. In various examples, the I/O components <NUM> may include user output components <NUM> and user input components <NUM>. User output components <NUM> may include, for example, display components for displaying information (for example, a liquid crystal display (LCD) or a projector), acoustic components (for example, speakers), haptic components (for example, a vibratory motor or force-feedback device), and/or other signal generators. User input components <NUM> may include, for example, alphanumeric input components (for example, a keyboard or a touch screen), pointing components (for example, a mouse device, a touchpad, or another pointing instrument), and/or tactile input components (for example, a physical button or a touch screen that provides location and/or force of touches or touch gestures) configured for receiving various user inputs, such as user commands and/or selections.

In some examples, the I/O components <NUM> may include biometric components <NUM>, motion components <NUM>, environmental components <NUM>, and/or position components <NUM>, among a wide array of other physical sensor components. The biometric components <NUM> may include, for example, components to detect body expressions (for example, facial expressions, vocal expressions, hand or body gestures, or eye tracking), measure biosignals (for example, heart rate or brain waves), and identify a person (for example, via voice-, retina-, fingerprint-, and/or facial-based identification). The motion components <NUM> may include, for example, acceleration sensors (for example, an accelerometer) and rotation sensors (for example, a gyroscope). The environmental components <NUM> may include, for example, illumination sensors, temperature sensors, humidity sensors, pressure sensors (for example, a barometer), acoustic sensors (for example, a microphone used to detect ambient noise), proximity sensors (for example, infrared sensing of nearby objects), and/or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components <NUM> may include, for example, location sensors (for example, a Global Position System (GPS) receiver), altitude sensors (for example, an air pressure sensor from which altitude may be derived), and/or orientation sensors (for example, magnetometers).

In some examples, the communication components <NUM> may detect identifiers or include components adapted to detect identifiers. For example, the communication components <NUM> may include Radio Frequency Identification (RFID) tag readers, NFC detectors, optical sensors (for example, one- or multi-dimensional bar codes, or other optical codes), and/or acoustic detectors (for example, microphones to identify tagged audio signals). In some examples, location information may be determined based on information from the communication components <NUM>, such as, but not limited to, geo-location via Internet Protocol (IP) address, location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and/or signal triangulation.

While various embodiments have been described, the description is intended to be exemplary, rather than limiting, and it is understood that many more embodiments and implementations are possible that are within the scope of the embodiments. Although many possible combinations of features are shown in the accompanying figures and discussed in this detailed description, many other combinations of the disclosed features are possible. Any feature of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment unless specifically restricted. Therefore, it will be understood that any of the features shown and/or discussed in the present disclosure may be implemented together in any suitable combination. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.

While the foregoing has described what are considered to be the best mode and/or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.

The scope of protection is limited solely by the claims that now follow. That scope is intended and should be interpreted to be as broad as is consistent with the ordinary meaning of the language that is used in the claims when interpreted in light of this specification and the prosecution history that follows and to encompass all structural and functional equivalents. Notwithstanding, none of the claims are intended to embrace subject matter that fails to satisfy the requirement of Sections <NUM>, <NUM>, or <NUM> of the Patent Act, nor should they be interpreted in such a way. Any unintended embracement of such subject matter is hereby disclaimed.

It will be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. Relational terms such as first and second and the like may be used solely to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "a" or "an" does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claim 1:
A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
receiving a first audio input from a user comprising spoken content, wherein the spoken content comprises textual content, one or more commands to be performed by an application on the textual content, or both;
receiving usage information from the application indicative of user interactions with the application prior to receiving the first audio input, while receiving the first audio input, or after receiving the first audio input, the usage information identifying a type of the application used by the user;
analyzing the first audio input using one or more natural language processing models to produce a first textual output comprising a textual representation of the first audio input;
analyzing the first textual output and the application usage information using one or more machine learning models to determine first context information of the first textual output wherein the first context information of the first textual output provides an indication of whether the first textual output includes the command and an indication of how the user intended to apply the command to content in the application; and
processing the first textual output in the application based on the first context information.