Context-based detection of end-point of utterance

Detecting an end-point of user's voice command or utterance with high accuracy is critical in automatic speech recognition (ASR)-based human machine interface. If an ASR system incorrectly detects an end-point of utterance and transmits this incomplete sentence to other processing blocks for further processing, it is likely the processed result would lead to incorrect interpretation. A method includes selecting a first semantic network based on context of the audio signal and more accurately detecting the end-point of user's utterance included in the audio signal based on the first semantic network and also based on at least one timeout threshold associated with the first semantic network.

The present disclosure is generally related to audio signal processing technology. More specifically, the present disclosure relates to automatic sound recognition method and system allowing improved user experience in one or more sound recognition applications.

II. DESCRIPTION OF RELATED ART

Speech recognition or automatic speech recognition (ASR) allows a user to control electronic devices via the user's voice command. ASR system takes an audio input signal which includes a voice command by a user, and aims to identify the voice command automatically. The identified voice command may be further processed by other signal processing blocks. For example, the identified voice command can be fed to a natural language understanding (NLU) block for further analysis.

ASR has been commercially deployed for many decades in various computing devices such as smartphones, tablets, and personal computers for its convenience over other interface methods. Conventional ASR, however, has limitation of understanding only small sets of keywords, which made it difficult for a user to communicate with devices in multi-turn conversations. Thanks to development in audio signal processing technology and recent breakthrough in machine learning technology such as Deep Neural Network (DNN) or Deep Learning Algorithms, ASR system is capable of understanding voice commands with more accuracy and with more flexibility allowing interactive voice response (IVR) communication or multi-turn conversations.

Detecting end-point of user's voice command or utterance with accuracy is critical in ASR-based human machine interface. When used in IVR communication or multi-turn conversations, detecting the end-point of the voice command or utterance is even more important. For example, a user may want to order a pizza via a ASR system by placing voice command of “Can I have a large . . . ” followed by few seconds of silence deciding which types of pizza the user wants to order. If the ASR system interprets the silence as the end-point of utterance and transmits this incomplete sentence to other processing blocks, located either in a local device or in a cloud via a communication network, for further processing, it is likely the processed result would lead to incorrect interpretation, posing a potential risk of throwing away the already spoken command (“Can I have a large . . . ”) out of conversation.

In a particular aspect, a method includes receiving, by an automatic speech recognition (ASR) module, an audio signal representing an utterance, and selecting a first semantic network, which includes a plurality of slots, based on context of the audio signal. The method includes performing, by the ASR module, ASR processing on a first portion of the audio signal to generate a first ASR output. The method further includes determining, by a natural language understanding (NLU) module, the first ASR output corresponds to an incomplete sentence based on the first semantic network, and, in response to determination that the first ASR output corresponds to the incomplete sentence, increasing a first timeout threshold associated with the first semantic network.

In another particular aspect, a device is configured to receive, by an automatic speech recognition (ASR) module, an audio signal representing an utterance, and select a first semantic network, which includes a plurality of slots, based on context of the audio signal. The device is configured to perfor, by the ASR module, ASR processing on a first portion of the audio signal to generate a first ASR output. The device is configured to determine, by a natural language understanding (NLU) module, the first ASR output corresponds to an incomplete sentence based on the first semantic network, and increase a first timeout threshold associated with the first semantic network, in response to determination that the first ASR output corresponds to the incomplete sentence.

In another particular aspect, an apparatus includes means for receiving, by an automatic speech recognition (ASR) module, an audio signal representing an utterance, and means for selecting a first semantic network, which includes a plurality of slots, based on context of the audio signal. The apparatus includes means for performing, by the ASR module, ASR processing on a first portion of the audio signal to generate a first ASR output. The apparatus further includes means for determining, by a natural language understanding (NLU) module, the first ASR output corresponds to an incomplete sentence based on the first semantic network, and means for increasing a first timeout threshold associated with the first semantic network, in response to determination that the first ASR output corresponds to the incomplete sentence.

In another particular aspect, a computer-readable storage device stores instructions that, when executed by a processor, cause the processor to perform operations receiving, by an automatic speech recognition (ASR) module, an audio signal representing an utterance, and selecting a first semantic network, which includes a plurality of slots, based on context of the audio signal. The operations also include performing, by the ASR module, ASR processing on a first portion of the audio signal to generate a first ASR output. The operations further include determining, by a natural language understanding (NLU) module, the first ASR output corresponds to an incomplete sentence based on the first semantic network, and increasing a first timeout threshold associated with the first semantic network, in response to determination that the first ASR output corresponds to the incomplete sentence.

Other aspects, advantages, and features of the present disclosure will become apparent after review of the application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.

V. DETAILED DESCRIPTION

Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It may be further understood that the terms “comprise”, “comprises”, and “comprising” may be used interchangeably with “include”, “includes”, or “including.” Additionally, it will be understood that the term “wherein” may be used interchangeably with “where.” As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

In the present disclosure, terms such as “determining”, “calculating”, “shifting”, “adjusting”, etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “generating”, “calculating”, “using”, “selecting”, “accessing”, and “determining” may be used interchangeably. For example, “generating”, “calculating”, or “determining” a parameter (or a signal) may refer to actively generating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, or accessing the parameter (or signal) that is already generated, such as by another component or device. The term “based on” (as in “A is based on B”) is used to indicate any of its ordinary meanings, including the cases (i) “based on at least” (e.g., “A is based on at least B”) and, if appropriate in the particular context, (ii) “equal to” (e.g., “A is equal to B”).

Unless indicated otherwise, any disclosure of an operation of an apparatus having a particular feature is also expressly intended to disclose a method having an analogous feature (and vice versa), and any disclosure of an operation of an apparatus according to a particular configuration is also expressly intended to disclose a method according to an analogous configuration (and vice versa).

In the present disclosure, systems and devices operable to detect end-point of utterance of a user's voice command are disclosed. In some implementations, detecting the end-point of utterance is based on context of the speech content including the utterance, as described further herein.

Detecting end-point of a user's utterance or voice command is critical to improve the accuracy of automatic speech recognition (ASR) system and to enhance user experience of using the ASR system. In one embodiment, the end-point detection may be implemented based on simple method of measuring silence period during a user's utterance. For example, a user may want to order a pizza by placing the following complete voice command (“Can I order a large thin crust pizza with a pineapple, an onion, and a green pepper?”) to an ASR system. In a first scenario where the user takes too much pause or delay between “large” and “thin” (e.g., “Can I order a large [long silence] thin . . . ?”), the ASR system may determine the long silence period comes immediately after the “large” as the end-point of the user's voice command. In this example, the partial voice command (“Can I order a large”) recognized by the ASR system lacks information about the item the user wants to order (e.g., pizza). Thus, the ASR system output based on this incomplete voice command, if processed by subsequent processing blocks without checking completeness of the voice command, may likely be quite different from the user's original expectation. In some implementations, if the ASR system is equipped with sentence completion checking mechanism, the ASR system, upon recognizing the voice command is incomplete, may prompt a follow up question asking the user for a confirmation, or asking the user to give another voice command for any missing information. In another implementation, the ASR system may require the user to speak the entire voice command all over again.

In a second scenario where a user takes too much pause or delay between “pizza” and “with” (e.g., “Can I order a large thin crust pizza [long silence] with . . . ?”), the ASR system may determine the long silence period comes immediate after the “pizza” as the end-point of the user's utterance. In this example, the partial voice command (“Can I order a large thin crust pizza”) recognized by the ASR system failed to capture user's entire utterance as a single command but nonetheless it still includes information about the item the user wants to order (e.g., pizza). The ASR system output based on this incomplete voice command, if processed by subsequent processing, may or may not likely generate result that the user can accept it. However, whether the result was accepted or rejected by the user, it is obvious that the user's experience of interacting with the ASR system is less than ideal because the remaining voice command portion after the long silence (“with a pineapple, an onion, and a green pepper”) was not processed by the ASR system as a single complete voice command with already processed partial voice command (“Can I order a large thin crust pizza”).

Additionally, user's voice command may be preceded by a special keyword such that a target virtual assistant (e.g., Amazon's Alexa, Google's Assistant, Apple's Siri, and so on) can awake out of low-power state or alternatively be ready for processing of subsequent voice command. This special keyword may be pre-defined for each specific virtual assistant. Non-limiting examples of this pre-defined keyword may include “Hey Snapdragon,” “Hey Siri,” “Alexa,” “Okay Google,” and so on. Alternatively, this special keyword may be any keywords defined by a user (e.g., user-defined keyword).

FIG. 1illustrates an example of an automatic speech recognition (ASR) system100for detecting end-point of utterance. The system100comprises automatic speech recognition (ASR) module130and natural language understanding (NLU) module140. The system100may further include various other processing blocks such as audio interface (I/O) module160, memory170, memory interface module (not shown), communication interface module180, and so on.

The audio I/O module160may include an audio front-end (AFE) module (not shown). The AFE module may include a digital-to-analog converter (DAC) and an analog-to-digital converter (ADC). The audio I/O module160may include at least one speaker and at least one microphone (a microphone array), not shown inFIG. 1. The at least one speaker and the at least one microphone may be coupled to DAC and ADC, respectively. The audio I/O module160may receive analog signals from the at least one microphone, convert the analog signals to digital signals using the ADC, and provide the digital signals to the ASR module130for automatic speech recognition processing. In some implementations, the NLU module140may provide digital signals to the audio I/O module160. The audio I/O module160may convert the digital signals to analog signals using the DAC and may provide the analog signals to the at least one speaker. In a particular implementation, the at least one speaker may include audio headset coupled to the system100and the at least one speaker may be coupled to the system100via wireless connections such as Bluetooth or Wi-Fi.

The system100may include a communication interface module180coupled, via a transceiver (e.g., a transmitter, a receiver, or both), to an antenna, which is not shown inFIG. 1. The system100may include the memory170, such as a computer-readable storage device. The memory170may include instructions, such as one or more instructions that are executable by the processor120to perform one or more of the techniques described further herein.

In a particular implementation, the ASR module130and NLU module140may be implemented by a single processor120or by a separate processor among a plurality of processors in the system100. Alternatively, the NLU module140may be configured to be located outside a local device110and may be coupled to the ASR module130via the communication interface module180. In another implementation, both the ASR module130and NLU module140may be configured to be located outside the local system100, and coupled to the processor120via the communication interface module180.

[ASR Module] When a user115of an electronics device110makes an utterance including a voice command, the ASR system100may recognize the utterance including the user's voice command, take actions in response to the recognized voice command, and present the result of the actions back to the user115. The ASR system100may be implemented on a local device110such as a mobile device, a smartphone, a tablet, an embedded system, or a personal computer. Alternatively, the ASR system100may be implemented on a remote cloud system wherein at least one server coupled to another server through various network connections.

The ASR module130may receive an audio signal representing the utterance by the user115and may extract sound features from the received audio signal. The sound features, which is also frequently referred to as audio fingerprints, may include acoustic parameters characterizing the voice command (or utterance) of the audio signal. For instance, non-limiting examples of these sounds features may include pitch, formant, Mel-Frequency Cepstrum Coefficients (MFCC), zero-crossings, signal energy level, spectral tilt, linear predictive coding (LPC) coefficients, and a plurality of gains. The ASR module130may extract sound features from small chunks of the audio signal at every fixed time interval such as a frame or a subframe. For example, a typical frame size may be 10-20 msec and a typical subframe size may be 2.5-5 msec. The ASR module130may compare these extracted sound features with at least one sound model (or acoustic model) to identify candidate matching phonemes.

In one embodiment, the ASR module130may identify N number of best matching candidate(s) based on confidence score. The confidence score may be obtained by combining or accumulating confidence score for each individual phoneme. The ASR module130may compare this confidence score with a confidence threshold value to identify N-best matching phonemes. The confidence threshold value may be pre-determined or dynamically adapted depending on user's preference, prior history of utterances, or types of applications.

The ASR module130may further compare the identified phonemes with at least one language model to identify candidate matching keywords with improved accuracy. Exemplary language models may include acoustic models (or sound models) and language model. For example, acoustic models may be used to estimate the most likely acoustic matching based on the speech features of the audio signal. Language models may be used to estimate the probability for the speech features based on the knowledge of language and linguistic structure thereof.

Sound models (or acoustic models) and language models may be generated based on trainings of sample speech data. These trainings may be based on template matching techniques or stochastic approaches. For example, Hidden Markov Model (HMM) is based on stochastic method and models a phoneme with a probability distribution, usually Gaussian Mixture Model (GMM). With the introduction of HMM in automatic speech recognition, speech recognition problem may be reduced to statistical optimization problem where the goal of the optimization is to find best matching sequence of words, given audio input signal representing utterance and various models such as acoustic and language models.

The ASR module130may send the result of ASR processing to other blocks in the downstream processing of the ASR result. In some implementations, the other processing blocks may include NLU module140, dialogue manager module260, natural language generator (NLU) module270, text-to-speech (conversion) module280, or any other blocks that may be used to take an action in responsive to the recognized voice command. The ASR module130may send ASR processing result to other blocks at regular time intervals This time interval may be pre-determined or may be dynamically adjusted during the ASR processing. In some implementations, the ASR module130may send ASR processing result to other blocks when a certain condition is satisfied.

[NLU Module] The NLU module140may include a module for traditional natural language processing (NLP) as well as other components such as a processor, storage, other processing modules, etc. In accordance with the disclosure, the NLU module140may perform semantic interpretation operation based on recognized text from the ASR module130and produce semantic information. Semantic information may indicate some value associated with the content of speech or recognized output (or text) from the ASR module130. For example, semantic tagging is a particular example of semantic interpretation operation. Semantic tagging is a process of recognizing specific meaning of words from an output of ASR processing, which is preferably performed by the ASR module130, and of assigning a tag to those words, whereby the tag is a classification of the associated word. Semantic interpretation processing may generate semantic information based on semantic networks. More detailed examples of semantic networks are presented below with reference toFIGS. 3-7. Each tag may be associated with a corresponding slot or net in semantic networks. It is generally known that some words in a phrase may be considered less important and those words may not receive a tag or may be given a special catch-all tag.

Although it is quite common that the NLU module140may perform semantic interpretation, skilled in the art would understand it is also feasible for the ASR module130, solely or in combination with the NLU module140, to perform semantic interpretation and produce semantic information, such as semantic tags. If the ASR module130is at least partially responsible for semantic tagging, an ASR output produced from the ASR module130may include not only recognized texts but also semantic information associated with the text.

In one embodiment, end-point detection may be based on detecting the number of silence frames or the duration of non-speech activity. Silence frames may refer to the frame that does not include user's voice command or the frame that only includes background noise or ambient sounds. Voice activity detector (VAD) or any other known methods may be used to identify whether a particular frame is a silent frame or non-speech frame. For example, The ASR system100, or preferably the NLU module140may detect the end-point by monitoring and/or tracking the number of consecutive silence frames. This scheme may be referred to as timeout (TO)-based end-point detection mechanism. The ASR system100, or preferably the NLU module140may keep track of lapsed time (or “silence period”) from the starting of the latest silence frame or alternatively from the ending of a user's last talk spurt. The lapsed time may be cleared or initialized at the beginning of new talk spurt. This lapsed time (e.g., silence or pause time from the end of a user's last talk spurt to the beginning of the new talk spurt) may be compared with a timeout threshold. If the lapsed time for the current silence period is greater than a timeout threshold, then the ASR system100or preferably the NLU module140may determine that an end-point of a user's voice command is detected, and may proceed with ASR processing for the audio signal corresponding to the recognized user's voice command. If the lapsed time for the present silence period (or latest silence period) is less than a timeout threshold, then the ASR system100or preferably the NLU module140may determine that an end-point is not detected. In such case, the ASR system100, or preferably the NLU module140may continue to increase the lapsed time until a user's new talk spurt is detected. The ASR module130or the NLU module140may clear or initialize the lapsed time upon detecting the current frame contains a user's talk spurt.

The timeout threshold may be set to a predetermined value or may be dynamically adjusted based on context of a user's voice command. The context information may be obtained by the ASR module130. Or alternatively context information may be obtained by the NLU module140and the context information may be provided back to the ASR module130for further processing. As a non-limiting example for the case where the timeout threshold is having a predetermined value, the timeout threshold may be set to either 600 msec or 30 consecutive silence frames (assuming each frame corresponds to 20 msec length), the ASR system100or preferably the NLU module140may detect end-point of a user's voice command when lapsed time exceed the timeout threshold. If the timeout threshold to determine end-point of a user's voice commend is set too small (e.g., 20-60 msec), then the user's entire utterance possibly intended to be a single voice command may not be captured as a single voice command. In such case, an ASR processing result may likely be rejected by the user. If the timeout threshold is set too large (e.g., 5-10 sec), then the ASR system may introduce large delay hindering a free-flowing conversational mode interaction.

Alternatively, the timeout threshold may be dynamically adjusted. In one embodiment, the timeout threshold may be adjusted based on multiple factors. As non-limiting examples, these factors may include at least one among a minimum timeout, a maximum timeout, semantic information of a user's voice command, historical data derived from the user's prior data, or any combination thereof.

In one embodiment according to the disclosure, the timeout threshold may vary depending on semantic analysis of recognized voice command. For example, semantic analysis may be performed by the ASR module130or by the NLU module140. The semantic analysis may be based on semantic networks. Alternatively, the semantic networks may also be referred to as a grammar file. The semantic networks may be the collection of a plurality of semantic network and each semantic network may include a plurality of slots or nets. For example, semantic networks may be Recursive Transition Network (RTN). In some implementation, the NLU module140may select at least one semantic network among a plurality of semantic networks. For example, the NLU module140may select a first semantic network among a plurality of semantic networks based on context information of audio signal. For example, context information may indicate that a user is about to place an order for online shopping. Additionally, context information may further indicate which types of specific shopping the user wants to engage in (e.g., ordering a pizza, purchasing flight tickets, or purchasing concert tickets). Context information may be obtained after parsing some portion of ASR output according to predetermined grammar or syntax. Context information may be obtained either by the ASR module130, or by the NLU module140after parsing some portion of ASR output according to predetermined grammar or syntax.

The ASR system100may have a separate semantic network corresponding to each business category or product category. For instance, the ASR system100may have a first semantic network for pizza shopping and a second semantic network for a flight ticketing. Alternatively, or additionally, the ASR system100may have a separate semantic network corresponding to a separate command indicating different actions. Examples of such actions may include “Open YouTube,” “Call Anthony,” “Set a reminder tomorrow morning.” In some implementations, each semantic network may have a different timeout threshold value pre-assigned per each slots or nets thereof. More examples and explanations about semantic networks are presented herein with reference toFIGS. 3-7.

The ASR module130and the NLU module140may be connected to other processing components160,170,180via a bus150. The ASR module130may receive the audio signal from the audio interface module160or communication interface module180. The ASR module130may output an ASR result to the audio interface160or to another local or cloud devices via communication interface module180. Although the memory170is depicted inside a mobile device110inFIG. 1, it should be understood that the memory170may be alternatively located in a cloud (not shown) or, additionally, may be located within either the ASR module130or the NLU module140. The memory170may store information required for ASR processing or NLU processing. In some implementations, this information may include, as a non-limiting example, acoustic model, language model, semantic networks, and program instruction for ASR and NLU processing.

FIG. 2illustrates another particular example of automatic speech recognition (ASR) system100for detecting end-point of utterance.FIG. 2includes the ASR module230, the NLU module240, and other downstream processing block250. For example, the other downstream processing block250may further include dialogue manager (DM)260, natural language generator (NLG)270, and text-to-speech (TTS) conversion module280. The ASR module230and the NLU module240may be similar to the ASR module130and the NLU module140as it was described with respect toFIG. 1. The ASR module230may generate an ASR output235and provide it to the NLU module240.

The NLU module140240may perform semantic interpretation operation, for example, such as semantic tagging process, based on recognized text from the ASR module130and produce semantic information. Based on the semantic information, the NLU module140240may determine whether end-point of an utterance is detected. The NLU module140240, in response to determination the end-point of an utterance is detected, may generate an NLU output245and provide it to further downstream processing block250. For example, an NLU output245may be provided to the dialogue manager260for additional processing.

Dialogue manager (DM)260may be a component of a complex dialog system and it generally determines what actions need to be taken based on the flow of conversation. The input to the DM245may be system-specific semantic representation of utterance produced by the NLU module140240. For example, in a flight-ticketing dialog system, the input may look like the following:

“TICKET_ORDER(From=“San Diego,” To=“Philadelphia,” Date=“2/13/2018”).” The DM260usually maintains state information, such as the dialog history, the latest unanswered question, etc. State information may enable a dialog system to interact with a user more naturally. For example, in an application where several answers are possible to a particular voice command, the DM260may select the best answer for the voice command based on a certain rule. However, if, state information, which is maintained based on the prior dialog history, shows the best answer was already used, then the DM260may select the 2ndbest answer to the voice command. The output of the DM260may be a list of instructions to other parts of dialog system, usually in a semantic representation. For example, in a flight-ticketing dialog system example above, the output of the DM260may be as follows: “TELL(Flight-Num=“KE202,” Flight-Time=“7:40”).” This semantic representation is usually converted to human language by the natural language generator (NLG)270.

Natural language generator (NLG)270has existed for a long time but commercial NLG technology has only recently become widely available in many applications such as smart speakers or virtual assistant applications. NLG270may generate natural language in machine representation system based on a knowledge base or a logical form. The typical processing of NLG270may be viewed as the opposite of the processing by the NLU module140240. For example, one of the objectives of the NLU module140240is to disambiguate an input sentence (or voice command) to produce machine representation language (or semantic representation) whereas one of the objectives of NLG270is to make decisions about how to put a concept represented by machine language (or semantic representation) into words or texts that may be presented to human through text-to-speech conversion.

Text-to-speech (TTS) conversion module280converts words or texts processed by a dialog system and represented by machine language (or semantic representation) into artificially generated speech. For example, TTS conversion module280receives the natural language generated by NLG270as an input and converts it into synthesized speech. TTS conversion module280may be one of many commercially known speech synthesizers and may be implemented by either hardware or software. In some implementations, TTS conversion module280may be comprised of multiple parts responsible for various processing routinely performed in synthesizing speech signal such as pre-processing, text normalization, text-to-phoneme conversion, and waveform generation.

The NLU module140240may provide a feedback signal290to the ASR module130230. As described above, the feedback signal290may include context information estimated by the NLU module140240. For example, the feedback signal290may include the context information indicating the context of a user's utterance (or voice command) is product purchasing (e.g., pizza order, flight reservation, or movie ticket reservation). Alternatively, or additionally, the feedback signal290may include semantic information of a user's utterance (or voice command) produced by the NLU module140240as described above. For example, the semantic information may indicate a particular semantic network selected based on the product purchasing context such as pizza order, flight reservation, or movie ticket reservation. The semantic information may also include tagging status associated with at least one net or slot in the selected network. In some implementation, the context information or semantic information in the feedback signal290may be used to detect an end-point of a user's voice command. For example, the context information or semantic information in the feedback signal290may be used to adjust at least one timeout threshold associated with any particular slot or net of a semantic network as described below with respect toFIG. 7. In some implementation, the context information or semantic information in the feedback signal290may also be used for the ASR module130230to detect the end-point of a user's voice command more accurately by comparing at least one timeout threshold of the selected semantic network with the duration of the latest silence period.

FIG. 3shows an exemplary semantic network pertaining to a particular context (e.g., pizza order) in accordance with the present disclosure. Semantic networks are the collection of each semantic network. Semantic networks may be referred to as a “gramma file,” and each semantic network in a gramma file may be frequently referred to as a “frame” in natural language understanding application. A gramma file may be stored in the memory170, and may be accessible by the ASR module130230or the NLU module140240. A frame in a gramma file is a slot-based or net-based network and may be used, by the NLU module140240, to determines whether a user's voice utterance constitutes a complete sentence or not.

In a preferred embodiment, a gramma file may comprise a plurality of “frame (or semantic network)” such that each of the frame corresponds to a particular context information of input audio. For example, context information may indicate the input audio corresponds to “business context” or “personal context.” Context information may further provide finer granularity. For example, context information indicating the input audio corresponding to business context may further provide specific type of business context. For instance, a gramma file (or semantic networks) may have a first semantic network for pizza shopping and a second semantic network for a flight ticketing. Alternatively, or additionally, a gramma file (or semantic networks) may have a separate semantic network corresponding to each of commands indicating different actions. For example, these actions may include “Open YouTube (“OPEN an application”),” “Call Anthony (“CALL someone”),” “Set a reminder tomorrow morning (“CHANGE setting”).”

Each frame or semantic network may include a plurality of slots or nets. A net or slot may be a placeholder for at least one input word or phrase. All the slots or nets in a single frame, if put together, may be sufficient to convey the meaning of a sentence. Each frame may contain a plurality of compulsory net(s) for some words in a phrase that may be important or necessary to complete a sentence, or additionally optional net(s) for some words that may be considered less important in the phrase. A frame may additionally include special nets, for example, such as “Start,” “And,” “End,” and a special catch-all tag.

Returning toFIG. 3, each box or circle inFIG. 3may correspond to a net or a slot in a particular frame300for food item (e.g., pizza) ordering context. The frame300includes compulsory nets such as [QUANTITY?]330and [ITEM?]350, and optional nets such as [WANT?]320, [SIZE?]340, and [TOPPING?]360. The frame300additionally includes special nets such as Start net310, End net390, and And net370. For example, Start net310may be used to indicate the beginning of a new sentence, and End net390may be used to indicate the completion of the sentence. The And net370may be a special net indicating that a user's voice command is a semantically complete sentence (e.g., all compulsory nets have been “filled”) but the user continues to speak more and thus there is a need to create another frame to capture a new voice utterance (e.g., a second order, or a supplemental command to the first order). In some implementations, the new frame may have the same frame structure as that of the previous frame, or may have a different frame structure depending on whether context information change has been detected or not. For example, when a user's voice command includes two separate but complete sentences such as “Can I have two pizzas and one large portion of garlic bread?” the ASR system100, in response to detecting the presence of “and” and in response to the determination that the first sentence prior to “and” is complete sentence, may create two copies of the same frame300, one for each complete sentence. The ASR system10then may attempt to fill any untagged nets for each of the two frames independently by asking questions such as “What size and topping for pizzas?” and “Any toppings for garlic bread?”.

Compulsory nets are the ones that need to be “filled” or “tagged” before completion of a sentence. Semantic tagging may refer to the process of assigning a tag or a label to a particular net in response to determination specific meaning of words or a phrase associated with the particular net has been identified from a user's utterance. For example, for the frame300to traverse from Start net310to the End net390(i.e., for the NLU module140240to determine if the recognized text from the ASR module130230is complete sentence), both [QUANTITY?] net330and [ITEM?] net350must be filled or tagged. Optional nets are the ones that tagging of them may be advantageous in understanding the meaning of a sentence but is not required to determine a user's utterance is a complete sentence (i.e., reach the End net390).

In one scenario, a user may want to order a pizza by placing the following a voice command (“Can I order a large thin crust pizza with a pineapple, an onion, and a green pepper?”) to an ASR system100. For illustration purpose, it is assumed the ASR system100may have already identified context information (e.g., pizza ordering conversation), and subsequently may have already selected a particular frame300based on the context information in a gramma file. Upon receiving the user's voice command, the ASR module130230in the ASR system100may perform ASR processing on the received user's voice command, and may generate the ASR output235(e.g., recognized text from the ASR processing). In some implementation, the ASR module130230may generate the ASR output235at regular time interval or when a certain condition is satisfied. If the NLU module140240successfully detects the end-point of user's utterance, the NLU module140240may generate the NLU output245such the entire voice command (“Can I order a large thin crust pizza with a pineapple, an onion, and a green pepper?”) is included in a single NLU output245. However, if a user takes too much pause or long silence between “pizza” and “with a pineapple, an onion, and a green pepper,” then it may be possible for the NLU module140240to determine the user's voice command ends immediately after “pizza,” in which case the NLU output245may include only a partial voice command (“Can I order a large thin crust pizza?”).

The ASR system100, or preferably the NLU module140240, may determine, based on the selected frame300, whether at least one recognized text received from the ASR module130230is semantically complete sentence by progressively analyzing portions of the at least one recognized text. For example, the semantic analysis or tagging process may start from an initial net (i.e., Start net310). Regarding the voice command (“Can I order a large thin crust pizza with a pineapple, an onion, and a green pepper?”), the ASR module130230may recognize an initial portion of the user's voice command, and may send a first recognized text (e.g., “Can I order”) to the NLU module140240at a first interval. Based on the first recognized text (“Can I order”), the NLU module140240may perform semantic tagging process and determine [WANT?] net320is filled or tagged (i.e., [WANT=“can I order”]). Then, the ASR module130230may recognize a next portion of the user's voice command, and may send a second recognized text (e.g., “a large thin crust pizza”) to the NLU module140240at a second interval. If the second recognized text (“a large thin crust pizza”) is received within a permitted time limit, the NLU module140240may continue to tag other nets in the frame300as follows: [QUANTITY=“a”], [SIZE=“large”], and [ITEM=“thin crust pizza”]. Finally, the ASR module130230may recognize the last portion of the user's voice command, and may send a third recognized text (e.g., “with a pineapple, an onion, and a green pepper”) to the NLU module140240at a third interval.

In this example, if the third recognized text was indeed received within a permitted time limit, the NLU module140240may complete tagging for [TOPPING?] net360as follows: [TOPPING=“with a pineapple, an onion, and a green pepper”]. Since all compulsory and optional nets have been tagged, the NLU module140240now can find the combination of all three recognized texts (i.e., the first, second, and third the recognized texts) may constitute a semantically complete sentence. In case the third recognized text was received outside a permitted time limit, the NLU module140240may not be able to tag [TOPPING?] net360based on the third recognized text in a timely manner. The NLU module140240, however, can still find the combination of the first and second the recognized texts may constitute a semantically complete sentence because at least all the compulsory nets (e.g., [QUANTITY=“a”]330, and [ITEM=“thin crust pizza”]350) have already been tagged, although one of the optional nets (e.g., [TOPPING?] net360) still remained untagged.

The frame300inFIG. 3is just an exemplary frame presented merely for the purpose of illustration. It should be understood by skilled in the art that various omissions, substitutions, or changes to the net configuration may be possible without departing from the spirit of the present disclosure. For example, any of the following nets such as [WANT?] net320, [SIZE?] net340, [TOPPING?] net360may be treated as compulsory net(s), or alternatively may be omitted from the frame300.

FIG. 4is an illustrative example of slots or nets for a semantic network or frame300pertaining to pizza order context. An ASR system100may store information400in the memory170indicating which words or phrases may be associated with which slots or nets in a particular frame in a gramma file. For example, the memory170may include information indicating that [WANT?] net420may be associated with any one of the words or phrases425such as “I want,” “I would like to,” “can I have,” and so on. In a similar token, the memory may further include information indicating which words or phrases are associated with [QUANTITY?] net430, [SIZE?]440, [ITEM?] net450, or [TOPPING?] net460. The information400may be stored in a form of look-up table (LUT) or in any other suitable data structure form suitable for indicating the connection between nets420430440450460of any particular frame and their corresponding candidate words or phrases425435445455465. The information400may also be stored in either a local memory or in a cloud memory connected via communication interface module180. The information400may be pre-determined based on samples of training data for any particular context information on which a particular frame is selected. In some implementation, the information400may be updated based on a user's prior history of utterance.

FIG. 5illustrates an example of another semantic network or frame500pertaining to another context (e.g. command indicating an “action”). Similar to the frame300for pizza order context inFIG. 3, the frame500includes special nets such as Start net510, End net590, and And net550. The frame500further includes compulsory nets like [ACTION?] net520and [OBJECT?] net530, and an optional net like [OPTIONS?] net540.FIG. 5shows information501indicating which words or phrases are associated with which slots or nets for the frame500pertinent to this context information. For example, an ASR system100may define some words or phrases indicating some action commands565, such as “add,” “call,” “play,” “open,” and “set,” to be associated with [ACTION?] net520560. Likewise, an ASR system100may also define various words or phrases indicating target(s)575of relevant action command(s)565to be associated with [OBJECT?] net530570.

In some implementations, the information501may also indicate which action commands can be semantically related with which type of objects. For example, “add” action command565may be used in conjunction with specific objects or targets such as “a reminder,” or “Anthony to the contact list”575, but not in conjunction with some other objects or targets575. In a similar manner, “call” action command565may only be used in conjunction with either name or phone number of a target person to be called575. In some implementations, the frame500may be a miscellaneous frame or a catch-all frame intended to capture for various actions related with any behavior change in any stage of processing.

FIG. 6illustrates another exemplary semantic network or frame600pertaining another context (e.g., purchasing a flight ticket context). As it was already described above, an ASR system100may select the frame600among a plurality of frames in a grammar file based on identified context information. The exemplary context information on which the frame600is selected inFIG. 6may be any of the following contexts such as business context, purchasing context, ordering a product context, or more specifically purchasing a flight ticket context. Similar to the frames inFIGS. 3 and 5, the frame600includes special nets like Start net610, End net690, and And net610. The frame600further includes compulsory nets like [BOOK?] net620, [TO?] net630, and [DATE/TIME?] net650, and optional nets like [FROM?] net640and [OPTIONS?] net660. For example, the words or phrase associated with [OPTIONS?] net650may include “extra baggage,” or “in-flight meals.” The And net610, for example, may be used for booking a return ticket, booking a cab to an airport, requesting for special assistance at the airport, or any other things.

FIG. 6shows [FROM?] net640is categorized as optional nets whereas [TO?] net630is categorized as compulsory nets (i.e., [TO?] net630is located in the shortest path from Start net610to End net690). This difference may be based on presupposition that estimating the departure of the flight (e.g., [FROM?]640) is easier than estimating the destination of the flight (e.g., [TO?]630). For example, the departure may be estimated based on the nearest airport to a user's current location, or alternatively the departure may be estimated based on the previous flight information of a user. The frame600inFIG. 6is merely presented for the purpose of illustration. It should be understood that various omissions, substitutions, or changes to the net configuration may be possible without departing from the spirit of the present disclosure.

FIG. 7shows another example of a semantic network or frame700pertaining to pizza order context illustrating exemplary timeout threshold information. The frame700inFIG. 7has frame structure quite similar to that of the frame300inFIG. 3in the sense that it includes same compulsory nets, optional nets, and special nets. In addition, the frame700further shows each box of the frame700has its own timeout (TO) threshold. In some implementation, the timeout threshold may be pre-determined or may be dynamically adjusted. The timeout threshold may be determined based on trainings or semantic analysis of sample database. The timeout threshold may be adjusted based on multiple factors. As non-limiting examples, these factors may include at least one among a minimum timeout, a maximum timeout, semantic information of a user's voice command, historical data derived from the user's prior data, or any combination thereof. In another implementation, all nets in a frame may have same timeout threshold or preferably they may have different timeout thresholds.

Returning back toFIG. 7, [WANT?] net720has a timeout threshold of 2 seconds and the [SIZE?] net740has a timeout threshold of 4 seconds. Timeout threshold may be used for an ASR system100to detect end-point of utterance. The ASR system100, preferably the NLU module140240, may keep track of a silence period. In some implementation, the silence period may be the number of consecutive silence frames or time lapsed from the ending of a user's last talk spurt. The silence period may be compared with a timeout threshold. For example, the NLU module140240may compare the silence period with the timeout threshold associated with a current net or slot in a frame. When the NLU module140240starts semantic tagging process, the current net may be Start net701. As the NLU module140240progressively continues tagging other nets, the current net may be the net that was last tagged.

In a scenario where a user wants to order a pizza by placing the following voice command (“Can I have a large pizza?”) to an ASR system, let's assume the NLU module140240just successfully completed semantic tagging process for the entire voice command as follows: [WANT=“can I have”], [QUANTITY=“a”], [SIZE=“large”], and [ITEM=“pizza”]. Since both compulsory nets ([QUANTITY=“a”]730and [ITEM=“pizza”]750) have been tagged, the NLU module140240may be able to declare the voice command is a complete sentence. This tagging process for the voice command (“Can I have a large pizza?”) in view of timeout thresholds is explained step by step below.

First, the NLU module140240may start tagging process from Start net710at time T0. At this time, Start net710may be called as the current net. Let's assume the ASR module130230sends an ASR output235that includes a recognized text for a first portion of the voice command (e.g., “can I have”) to the NLU module140240at time T1. Then, the NLU module140240may be able to tag [WANT?] net720as follows: [WANT=“can I have”] because the phrase “can I have” is one of the phrases425associated with [WANT?] net320420720as presented inFIG. 4. At this time, the current net indicates or points to [WANT?] net720, and the timeout threshold of the current net (i.e., [WANT?] net720) becomes the current timeout threshold (i.e., 2 seconds) for the frame700.

Second, at T2, the ASR module130230sends an ASR output235that includes a recognized text for a second portion of the voice command (e.g., “a”) to the NLU module140240. T2 may be expressed as follows: T2=T1+ delta1, where delta1≥0. If delta1=0 (i.e., T2 is same as T1), it indicates the second portion of the voice command (e.g., “a”) was sent to the NLU module140240at the same time as the first portion of the voice command (e.g., “can I have”), possibly in same ASR output235. If delta1≠0 (i.e., T2 is different than T1), it indicates the second portion of the voice command (e.g., “a”) was sent to the NLU module140240after the first portion of the voice command (e.g., “can I have”) was sent to the NLU module140240.

The NLU module140240may keep track a silence period between the ending of the last talk spurt (“can I have”) and the starting of the new talk spurt. At T2, the silence period is the time between “can I have” and “a.” If the silence period is less than the current timeout threshold (i.e., 2 seconds), the NLU module140240may be able to tag the following net [QUANTITY?] net730with “a” as follows: [QUANTITY=“a”] based on the information400inFIG. 4. At this time, the current net indicates or points to [QUANTITY?] net730, and the timeout threshold of the current net (i.e., [QUANTITY?] net730) becomes the current timeout threshold (i.e., 3 seconds) for the frame700. If the silence period is longer than the current timeout threshold (i.e., 2 seconds), the NLU module140240may incorrectly determine an end-point of a user's voice command is detected, and may transmit the sentence that includes only “can I have” to other downstream processing block250as part of the NLU output245for further processing. Transmitting this seeming incomplete sentence to other downstream processing block is clearly incorrect interpretation of user's intent by the ASR system100, and it may pose risk of throwing off the entire natural conversion out of due course.

In accordance with the present disclosure, the NLU module140240may determine the sentence having only “can I have” as an incomplete sentence because the current net still points to [WANT?] net720, or because none of the two compulsory nets ([QUANTITY=“a”]730and [ITEM=“pizza”]750) have been tagged. The NLU module140240then alternatively may decide not to transmit the incomplete sentence to other downstream processing block250. The NLU module140240may instead ask the user a follow up question for more information about unflagged nets, for example, such as [QUANTITY?] net730and [ITEM?] net750. In another implementation, in response to determination that the silence period is longer than the current timeout threshold (i.e., 2 seconds) and in response to the determination that at least one of the two compulsory nets ([QUANTITY=“a”]730and [ITEM=“pizza”]750) is still untagged, the NLU module140240may augment or increase the current timeout threshold. For example, the NLU module140240may increase the current timeout threshold from 2 seconds to 3 seconds in which case there is higher probability the NLU module140240may be able to tag [QUANTITY?] net730based on the second portion of the voice command (e.g., “a”).

In a similar manner, the remaining voice command (“large pizza?”) may be tagged ([SIZE=“large”] and [ITEM=“pizza”]) by the NLU module140240so long as silence periods between words in the voice command do not exceed the current timeout threshold. In this particular example, the NLU module140240may progress tagging in the following order: [WANT=“can I have”]720, [QUANTITY=“a”]730, [SIZE=“large”]740, and [ITEM=“pizza”]750. Therefore, the current timeout threshold progressively changes from 2 seconds to 3 seconds, 4 seconds, and 0.5 second, which are timeout thresholds for [WANT?] net720), [QUANTITY?] net730, [SIZE?] net740, and [ITEM?] net750, respectively.

In another scenario for the same voice command (“Can I have a large pizza?”), let's assume a user takes really long pause (e.g., longer than 4 seconds) between “Can I have a large” and “pizza.” When the recognized text “pizza” is finally received from the ASR module130230, the NLU module140240may have already completed tagging for [WANT=“can I have”]720, [QUANTITY=“a”]730, and [SIZE=“large”]740. The current timeout threshold at this time would be 4 seconds, which is the timeout threshold for the last tagging net (i.e., [SIZE=“large”]740). Since the silence period is longer than the current timeout threshold (i.e., 4 seconds), the NLU module140240may determine it detect an end-point of user's voice command and may transmit the detected sentence (“can I have a large”) to other processing blocks although the detection of end-point is incorrect. Alternatively, in accordance with the present disclosure, the NLU module140240may check if there is any unflagged net(s) in the shortest path from the current net (i.e., [SIZE?] net740) to End net790. In this particular example, [ITEM?] net750is located in the shortest path, which means it may be a compulsory net, and it is still unflagged. Thus, the NLU module140240may follow up with a user by asking, for example, “what item are you looking for?”. In case the ASR system100maintain user's prior history of similar voice commands associated with the same or similar context (e.g., pizza ordering context, or product ordering context), the ASR system100may refine the follow up questions based on information from the prior history. For example, if the prior history indicates that the user frequently placed an order for ordering “pizza” or “garlic bread,” the follow question for [ITEM?] net750would likely be “would that be a large pizza or a large portion of garlic bread?” instead of “what item are you looking for?”.

In another implementation, the NLU module140240, in response to detect that [ITEM?] net750is still unflagged, may adjust or augment the current timeout threshold from 4 seconds to say 5 seconds. The prior history of context based conversations by a user may be used to determine the timeout threshold for each net. For example, let's assume the prior history, based on a number of prior conversations over a long period of time, shows that [ITEM?]750was usually followed by [TOPPING?]760, but [TOPPING?]760was usually not followed by anything. Since the frame700shows [ITEM?]750can go directly to End net790or [TOPPING?]760, the ASR system100may set the timeout threshold (e.g., 0.5 second) for [ITEM?]750to a slightly higher value (e.g., 0.25 second) than that of [TOPPING?]760. The prior history of context based conversations by a user may be used to update the timeout threshold for each net. For example, let's assume the prior history shows that a particular user tends to pause longer right after [ITEM?]750than after [SIZE?]740. In this case, [ITEM?]750may have a longer timeout threshold (e.g., 2 seconds instead of 0.5 second) and [SIZE?]740may have a shorter timeout threshold (e.g., 1 second instead of 3 seconds).

FIG. 8is a flow chart illustrating an example of a method of detecting the end-point of audio signal representing an utterance. The method800may be performed by a local device110ofFIG. 1. For example, the method800may be performed by a processor120including the ASR module130230or the NLU module140240.

The method800includes receiving, by an ASR module, an audio signal representing an utterance, at810. In a particular example, the ASR module130230may receive an audio signal including a user's command via an audio I/O module160. The method800includes selecting a first semantic network based on context of the audio signal, at820. In a particular example, either the NLU module140240or the ASR module130230may determine context of the input audio signal, and may select a particular semantic network or a frame among a gramma file based on the determined context information. A gramma file may be stored in the memory170and may be accessible to either the ASR module130230or the NLU module140240.

The method800includes performing, by the ASR module, automatic speech recognition processing on a first portion of the audio signal to generate a first ASR output, at830. In a particular example, the ASR module130230may perform ASR processing and generate at least one ASR output235at certain time interval. For example, the time interval may be fixed based on a timer (e.g., at every 500 msec), or may vary based on characteristics of the audio signal. The ASR module130230may transmit the at least one ASR output235to the NLU module140240for further processing.

The method800further includes determining, by a NLU module, the first ASR output corresponds to an incomplete sentence based on the first semantic network, at840. In a particular example, the NLU module140240may determine the ASR output received from the ASR module130230corresponds to an incomplete sentence. This determination840may be based on the first selected network selected at least partially based on the context information of the audio signal as was described with respect toFIGS. 3-7. For example, this determination840may be based on comparison of each silence period of nets of the first selected network with at least a particular timeout threshold associated with at least one nets of the first selected network.

The method800includes increasing a first timeout threshold associated with the first semantic network in response to determination that the first ASR output corresponds to the incomplete sentence at850. In a particular example, the NLU module140240may increase the timeout threshold for a current net (i.e., the net that was just tagged most recently) upon determining that at least one ASR output235received from the ASR module130230may correspond to incomplete sentence. For example, the NLU module140240may determine the ASR output235is incomplete (or incomplete sentence) when at least one compulsory net is still unflagged. The NLU module140240may also increase the timeout threshold for a current net at least partially based on a prior history of voice commands. The voice commands may be associated with the current semantic network

FIG. 9shows a flow chart illustrating another example of a method of detecting the end-point of audio signal representing an utterance. The method900may be performed by a local device110ofFIG. 1. For example, the method900may be performed by a processor120including the ASR module130230or the NLU module140240. In some implementation, the method900may follow the steps included in the method800inFIG. 8.

The method900includes performing, by the ASR module, ASR processing on a second portion of the audio signal to generate a second ASR output, at910. In a particular example, the ASR module130230may perform ASR processing and generate at least one ASR output235at certain time interval. For example, the ASR module130230may perform ASR processing on a second portion of the audio signal in response to the determination, by the NLU module140240, at least one previous ASR output235was incomplete sentence. For example, the NLU module140240may determine the ASR output235is incomplete (or incomplete sentence) when at least one compulsory net is still unflagged. The ASR module130230may transmit the at least one ASR output235to the NLU module140240for further processing.

The method900includes determining, by the NLU module, the second ASR output corresponds to a complete sentence based on the first semantic network, at920. In a particular example, the NLU module140240may determine at least one ASR output received from the ASR module130230corresponds to a complete sentence. This determination920may be based on the first selected network selected at least partially based on the context information of the audio signal as was described with respect toFIGS. 3-7. For example, this determination920may be based on whether all the compulsory net in the selected first semantic network is already flagged or not, or alternatively whether the current net during semantic tagging process successfully traversed to the End net390590690790.

The method900includes generating a first NLU output, in response to determination that the second ASR output corresponds to the complete sentence, at930. In a particular example, the NLU module140240may generate the NLU output245in response to determination at least one ASR output received from the ASR module130230corresponds to the complete sentence. The NLU module140240may transmit the NLU output245to other downstream processing block. In some implementation, the other downstream processing block may further include dialogue manager (DM)260, natural language generator (NLG)270, text-to-speech (TTS) conversion module280, or any other blocks that may be used to take an action in responsive to the NLU output245.

The method900includes initiating a first action to be executed on the electronic device, at940. In a particular example, the step940may be performed by a local device110ofFIG. 1. The action may include any action that may be reasonably anticipated in response to successfully recognized voice commend by the ASR system100. For example, this action may include “initiating a call” in response to the recognized voice command of “call Anthony,” or “launching music play application” in response to the recognized voice command of “play my favorite song.”

In particular aspects, the method800ofFIG. 8or the method900ofFIG. 9may be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a controller, another hardware device, firmware device, or any combination thereof. As an example, the method800ofFIG. 8or the method900ofFIG. 9may be performed by a processor that executes instructions, as described with respect toFIG. 10.

FIG. 10shows a block diagram illustrating a particular example of a device that is operable to perform automatic speech recognition. In various implementations, the device1000may have more or fewer components than illustrated inFIG. 10. In an illustrative example, the device1000may correspond to the system100and may operate according to the method ofFIGS. 8-9.

In a particular implementation, the device1000includes a processor1006(e.g., a CPU). The device1000may include one or more additional processors, such as a processor1010(e.g., a DSP). The processor1010may include ASR engine1091, NLU engine1092, or a combination thereof. For example, the ASR engine1091may be the ASR module130230, and the NLU engine1092may be the NLU module140240. As another example, the processor1010may be configured to execute one or more computer-readable instructions to perform the operations of the ASR engine1091or NLU engine1092. Thus, the CODEC1008may include hardware and software. Although the ASR engine1091or NLU engine1092are illustrated as components of the processor1010, in other examples one or more components of the ASR engine1091or NLU engine1092may be included in the processor1006, a CODEC1034, another processing component, or a combination thereof.

The device1000may include a memory1032and the CODEC1034. The CODEC1034may include a digital-to-analog converter (DAC)1002and an analog-to-digital converter (ADC)1004. A speaker1036, a microphone or a microphone array1035, or both may be coupled to the CODEC1034. The CODEC1034may receive analog signals from the microphone array1035, convert the analog signals to digital signals using the analog-to-digital converter1004, and provide the digital signals to the ASR engine1091. In some implementations, the ASR engine1091or the NLU engine1092may provide digital signals to the CODEC1034. The CODEC1034may convert the digital signals to analog signals using the digital-to-analog converter1002and may provide the analog signals to the speaker1036.

The device1000may include a wireless controller1040coupled, via a transceiver1050(e.g., a transmitter, a receiver, or both), to an antenna1042. The device1000may include the memory1032, such as a computer-readable storage device. The memory1032may include instructions1060, such as one or more instructions that are executable by the processor1006, the processor1010, or a combination thereof, to perform one or more of the techniques described with respect toFIGS. 1-7, the method ofFIGS. 8-9, or a combination thereof.

As an illustrative example, the memory1032may store instructions that, when executed by the processor1006, the processor1010, or a combination thereof, cause the processor1006, the processor1010, or a combination thereof, to perform one or more of the techniques described with respect toFIGS. 1-7, the method ofFIGS. 8-9, or a combination thereof.

The memory1032may include instructions1060executable by the processor1006, the processor1010, the CODEC1034, another processing unit of the device1000, or a combination thereof, to perform methods and processes disclosed herein. One or more components of the system100ofFIG. 1may be implemented via dedicated hardware (e.g., circuitry), by a processor executing instructions (e.g., the instructions1060) to perform one or more tasks, or a combination thereof. As an example, the memory1032or one or more components of the processor1006, the processor1010, the CODEC1034, or a combination thereof, may be a memory device, such as a random access memory (RAM), magnetoresistive random access memory (MRAM), spin-torque transfer MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, or a compact disc read-only memory (CD-ROM). The memory device may include instructions (e.g., the instructions1060) that, when executed by a computer (e.g., a processor in the CODEC1034, the processor1006, the processor1010, or a combination thereof), may cause the computer to perform at least a portion of the methods ofFIGS. 8-9, or a combination thereof. As an example, the memory1032or the one or more components of the processor1006, the processor1010, the CODEC1034may be a non-transitory computer-readable medium that includes instructions (e.g., the instructions1060) that, when executed by a computer (e.g., a processor in the CODEC1034, the processor1006, the processor1010, or a combination thereof), cause the computer perform at least a portion of the method ofFIGS. 8-9, or a combination thereof.

In a particular implementation, the device1000may be included in a system-in-package or system-on-chip device1022. In some implementations, the memory1032, the processor1006, the processor1010, the display controller1026, the CODEC1034, the wireless controller1040, and the transceiver1050are included in a system-in-package or system-on-chip device1022. In some implementations, an input device1030and a power supply1044are coupled to the system-on-chip device1022. Moreover, in a particular implementation, as illustrated inFIG. 10, the display1028, the input device1030, the speaker1036, the microphone array1035, the antenna1042, and the power supply1044are external to the system-on-chip device1022. In other implementations, each of the display1028, the input device1030, the speaker1036, the microphone array1035, the antenna1042, and the power supply1044may be coupled to a component of the system-on-chip device1022, such as an interface or a controller of the system-on-chip device1022. In an illustrative example, the device1000corresponds to a communication device, a mobile communication device, a smartphone, a cellular phone, a laptop computer, a computer, a tablet computer, a personal digital assistant, a set top box, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, an optical disc player, a tuner, a camera, a navigation device, a decoder system, an encoder system, a base station, a vehicle, or any combination thereof.

In the aspects of the description described above, various functions performed have been described as being performed by certain components or modules, such as components or module of the system100ofFIG. 1. However, this division of components and modules is for illustration only. In alternative examples, a function performed by a particular component or module may instead be divided amongst multiple components or modules. Moreover, in other alternative examples, two or more components or modules ofFIG. 1may be integrated into a single component or module. Each component or module illustrated inFIG. 1may be implemented using hardware (e.g., an ASIC, a DSP, a controller, a FPGA device, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

The steps of a method or algorithm described in connection with the aspects disclosed herein may be included directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transient storage medium known in the art. A particular storage medium may be coupled to the processor such that the processor may read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

The previous description is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein and is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.