Patent ID: 12236947

DETAILED DESCRIPTION

Referring toFIG.1, in one embodiment, a voice-based assistant100makes use of an audio input signal105, for example, picked up at a microphone101(e.g., an electrical and/or digital signal representing an acoustic signal). In general, the audio signal includes a speech signal (e.g., a “voice command” delimited by silence or non-speech input) representing speech produced by a speaker (the “user,” not shown in the figure) in the acoustic environment of the microphone (e.g., in a vehicle in which the microphone is mounted, in which case the user may be the driver or possibly a passenger). Optionally, as discussed later in this document, the system includes a “push-to-talk” button102that the user may press when speaking or starting to speak, a camera103monitoring the user's facial expressions, and other inputs associated with the user.

The assistant100is able to process voice commands that have a flexible format. The format is flexible on one or more of the following aspects. First, if the assistant has “wake-up word” (WUW) (e.g., a name given to the assistant), that WUW is not required to be at the start of the voice command, for example, with the WUW being permitted at alternative locations in the voice command or absent completely from the voice command. The name used for the WUW may be configurable for the assistant, and how the WUW is processed and/or required in a voice command may also be configurable to the assistant. Second, the system100may include or provide an interface to multiple separate “sub-assistants,” for example, in an in-vehicle system, such “sub-assistants” may include an assistant integrated in the vehicle, an assistant integrated into the user's smartphone, and a home assistant accessible over a communication link from the vehicle. When there are such multiple sub-assistants, each may have its own configuration governing the WUW used and requirements of how the WUW is located in a voice command for that assistant. Generally, with these capabilities, the system100can provide a flexible and easy-to-use voice interface to the assistant(s) supported by the system.

The system100includes a transcriber, which performs an automated speech-to-text conversion (i.e., “speech recognition”) of the audio signal105to produce a text output115representing a sequence of words (or a representation of a set of sequences, such as a list of N best hypotheses) spoken by the user. In some examples, a sequence of words may be accompanied by a score representing a quality of match of the sequence to the input audio or match of individual words to corresponding audio, and the sequence of words may be accompanied by times of occurrence of the words in the sequence permitting alignment of the words with the input audio. The transcriber is configured with configuration data160, which may include a language model and an acoustic model that is used by the transcriber for the conversion of audio to text. In some implementations, rather than waiting for receipt of an entire utterance in the audio signal before providing the text output, partial transcripts may be provided as the audio signal is acquired and processed by the transcriber. In this way, the system may be able to act with a lower delay (e.g., “latency”).

The system processes the audio signal during speech or potentially speech events. Not shown inFIG.1is a speech-activity detector that gates the audio signal105to only pass signals that have sufficient loudness (e.g., energy) and/or that have speech-like characteristics. Therefore, the system can be thought of as processing sections of input of varying duration triggered by the speech activity detector. Note that in some implementations, rather than gating the audio signal, similar speech activity detection functions may alternatively be integrated into the transcriber or other components that process an ungated audio signal.

The configuration data160is generally or largely determined before operation of the system. For example, an offline training/configuration component180uses an audio or transcribed text corpus of commands for one or more assistants to determine the structure of valid commands for those assistants, and a language model may be derived from the corpus for use in processing audio input containing speech. For example, the language model may be a statistical language model, meaning that different word sequences are associated with different probability or scores indicating their likelihood. In some cases, such a statistical language model is an “n-gram” model in which statistics on n-long sequences of words are used to build the model. In some cases, some or all of the language model may have a finite-state form, for example, specifying a finite set of well-structured commands. In some cases, such a finite-state form is statistical in that different commands may have different probabilities or scores. In some examples, a combination of approaches, such as n-gram and finite-state forms, are combined in a finite-state transducer. It should be understood that there are a number of alternatives to the form of the language model. In some examples, the occurrences of wake words and/or names given to assistants, or generic subjects that may not be explicitly defined (e.g., “computer,” “automobile,” “you”), are identified in the corpus and are essentially replaced with placeholders to permit configuration of different words or names for assistants, including coined names that may not actually occur in the corpus, for use in runtime systems without having to derive new language models. In some examples, the language model used to automatically transcribe an audio input may be biased to avoid missing true occurrences of wake words that are part of commands in the process of automated transcription of audio input. In some examples, the language model provides a way to tag output words, for example, to indicate that the word occurred in the position of a subject or assistant name in the language model, and such tags are used in further processing of the output text.

In some examples, the configuration data160may be determined in part during operation or shortly before operation of the system. For example, an optional online training/configuration component170may receive, from the user, a name they have given to an assistant. For example, the user may say or type the word “Sophie” to name the assistant. The configuration data160is then amended to modify the language model to permit the word “Sophie” in the name position in the language model and/or modify the pronunciation of a placeholder for the name with a pronunciation of the word “Sophie,” for example, determined by an automated text-to-phoneme converter (e.g., as is used in a speech synthesis system). In some examples, the online training/configuration component170may receive other information that is used in configuring components of the system, for example, the names of family members of the user and/or names of passengers in a vehicle. These names may be used to configure the language model to replace a placeholder for non-system names that may be present in utterances spoken by the user but that are not directed to the system.

The text output115may not represent a truly intended command. For example, the microphone may detect extraneous conversation between the user and other people in the environment such that the words spoken should not be acted upon. The system100represented inFIG.1includes a reasoner150, which includes logic, classifiers and/or models that are used to determine whether the input should be considered to have represented a command uttered (i.e., system-directed input) by the user or extraneous input (i.e., non-system-directed input). Note that in alternative embodiments, the function of the reasoner described below may be integrated into a dialog control component rather than being separate.

A variety of different logical implementations of the reasoner, or its function in a dialog control component can be implemented. The reasoner150has available to it a number of inputs that it processes to determine if the input is system-directed or not, and in cases where the system supports multiple different assistants, to which assistant a system-directed input is to be directed. As introduced above, one source of input is the text output115of the transcriber, which may be received incrementally as the transcriber processes the audio input105. Another source of input is the audio signal105itself, or possibly a processed version of the audio signal for example, representing a sequence of signal processing features (e.g., corresponding to fixed-length sections/“frames” of the audio signal, and/or utterance features, such as duration, pitch slope, etc.). As illustrated inFIG.1, the reasoner may optionally receive yet other inputs upon which it makes the system/non-system directed decision. For example, a camera103may provide an image of the user's face, and a “push-to-talk” button102may provide a signal indicating when the button is pressed.

As introduced above, the reasoner150processes both text output115and the audio signal105. The audio signal105is processed by an acoustic classifier152. In some implementations, this classifier is a machine learning classifier that is configured with data (i.e., from configuration data160) that was trained on examples of system-directed and of non-system directed utterance by an offline training system180. In some examples, the machine-learning component of the acoustic classifier152receives a fixed-length representation of the utterance (or at least the part of the utterance received to that point) and outputs a score (e.g., probability, log likelihood, etc.) that represents a confidence that the utterance is a command. For example, the machine-learning component can be a deep neural network. Note that such processing does not in general depend on any particular words in the input, and may instead be based on features such as duration, amplitude, or pitch variation (e.g., rising or falling pitch). In some implementations, the machine-learning component processes a sequence, for example, processing a sequence of signal processing features (e.g., corresponding to fixed-length frames) that represent time-local characteristics of the signal, such as amplitude, spectral, and/or pitch, and the machine-learning component processes the sequence to provide the output score. For example, the machine learning component can implement a convolutional or recurrent neural network.

One way the reasoner150may process the text output115is to determine whether WUW occurred at a prescribed location for the assistant (e.g., at the beginning of the voice command). The reasoner includes a WUW classifier153that is configured with the WUWs (i.e., names) given to each of the assistants (i.e., system-defined and/or user-defined WUWs) and processes the text output115to detect their presence. In addition to being configured with the WUWs, the WUW classifier is configured to determine whether the WUW occurred at a prescribed location for the assistant with which that WUW is associated. This determination may use a variety of techniques. In one such technique, the assistant may be configured to permit one or more of (a) the beginning of a command utterance, (b) at the end of the command utterance, and/or (c) within the utterance and the WUW classifier makes a binary decision of whether the WUW is at an allowable location. If it is, the WUW classifier outputs an indication of which assistant was identified by the WUW, and otherwise provides a negative or no output. In another technique, the context of the WUW in the text output is used to determine whether the WUW occurred at a prescribed position. One such context-based determination uses a trained classifier (e.g., a neural network, a decision tree, etc. trained on system-directed and non-system-directed text transcription), which uses the word context (e.g., a sequence of words before and after the WUW in the text output, or a function of the words such as their parts of speech) to determine a score of the WUW. Note that different assistants may have differently trained scorers for the location of the keyword.

Another way the reasoner150may process the text output115is using a text-based classifier156, which inputs the text output115, and outputs a score that indicates whether that word sequence represents a system-directed command. In some such examples, there may be multiple such classifiers, one trained for each of the assistants. In some examples, the text output is pre-processed, for example, replacing words with syntactic categories such as parts of speech, semantic classes such as person name or assistant name, and/or trained embedding such as contextualized word embeddings (e.g., BERT). The text output and/or its preprocessed form are then processed using the classifier to indicate whether the text output (or a partial text output available while the transcriber110is still processing the audio signal105and the user has not yet stopped speaking). InFIG.1, the text-based classifier156is indicated as optional with a dotted line. It should be understood that in general, the reasoner150may include the WUW classifier153, the text classifier156, some other text-based classifier, or any subset of one or more of such text-based classifiers.

In some examples, the reasoner implements a classifier that takes advantage of yet other input to make the classification. For example, video from the camera103monitoring the user's facial expressions, lip movement, etc., is provided to a visual classifier157of the reasoner150to aid in the classification. In a machine learning approach, such a video signal may be processed using a machine learning approach that is trained on video (i.e., image sequences) for users making system-directed and non-system-directed utterances. In some examples, the video is processed in conjunction with the audio, essentially combining functions of the acoustic classifier152and visual classifier157(e.g., to permit taking advantage of relationships between visual and audio cues that help determine when the user intends for an utterance to be system directed).

Another way that the reasoner150has evidence that the user intends for an utterance to be system directed is if they press the “push-to-talk” button102before they start to speak or during the interval that they were speaking.

In some alternative embodiments, the reasoner processes the audio input105with a conventional word spotter configured to detect particular words. For example, a Hidden-Markov Model (HMM) based word spotter may be used.

The reasoner150then combines the outputs of the various classifiers it has at its disposal (e.g., one or more of classifiers152-152,156-157). In some implementations, a combined classifier151users predetermined logic and/or arithmetic combinations of the scores of the classifiers to yield an overall determinization of whether the input (e.g. the audio, text, and possibly video and button) represent a system directed utterance. In some implementations, this classifier is itself trained on instances of system-directed and non-system-directed inputs, for example, being implemented as a decision tree, neural network, or other parameterized and configurable classifier. In general, each of the classifiers have scores or probabilities of the utterance or partial utterance being system directed associated with them, and the reasoner may essentially use such quantities to weigh the different factors. For example, a very strong indication that a WUW occurred at a required position may make up for a relatively lower score from the text or acoustic classifier, resulting in the reasoner declaring that the input was intended to be a voice command for the assistant. Conversely, because a WUW is not always required, a low score from the WIW classifier may be compensated for by a high score from the acoustic classifier or the text classifier.

Note that in a number of the techniques for processing inputs to the reasoner, the classifiers involved were configured based on training. Such training may be the same for all users of the system, but it should be understood that the configurations may be retrained to a particular user (e.g., the user that is known to be driving a vehicle) and/or adapted to that user during use by collecting that user's system directed and non-system-directed utterances. In such a way, the performance of the reasoner may be improved.

In some implementations, if WUW classifier153determines that a WUW was detected in isolation, the output of the reasoner may indicate to the relevant assistant that is should enter into a follow-up dialog, for example, causing a “how can I help you?” or “what can Mercedes do for you?” prompt. Note that in some situations, a degree of dialog control is implemented by the combination of the reasoner and the assistant in order to carry over the context in processing a next utterance to account for the higher likelihood that the next audio input is a voice command and for that command to be directed to the assistant identified by the WUW in the previous input.

In situations in which the reasoner150determines that an utterance is a system-directed command directed to a particular assistant, it sends a reasoner output155to one of the assistants140A-Z with which the system100is configured. As an example, assistant140A includes a natural language understanding (NLU)120, whose output representing the meaning or intent of the command is passed to a command processor130, which acts on the determined meaning or intent.

Various technical approaches may be used in the NLU component, including deterministic or probabilistic parsing according to a grammar provided from the configuration data160, of machine-learning based mapping of the text output115to a representation of meaning, for example, using neural networks configured to classify the text output and/or identify particular words as providing variable values (e.g., “slot” values) for identified commands. The NLU component120may provide an indication of a general class of commands (e.g., a “skill”) or a specific command (e.g., and “intent”), as well as values of variables associated with the command. The configuration of the assistant140A may use configuration data that is determined using a training procedure and stored with other configuration data in the configuration data storage160.

In some implementations in which there are multiple sub-assistants, the WUW for such an assistant may be detected. However, rather than relying on the transcription of the audio input when it is initiated by the reasoner, the assistant processes the original audio signal105(which may have been buffered to be able to “roll back” the input so that the assistant gets the entire utterance from the point when speech was detected). For example, if the WUW corresponds to a home assistant (e.g., “Alexa” or “Hey Google”), the audio input105may be transmitted to the home assistant for processing. For example, in a vehicle, the user may say “Alexa, open the garage door” and the word spotting output145may indicate the presence of “Alexa” at the start of the voice command, but the text output may not be correct and the original audio is provided to the Alexa assistant for processing to cause the garage door to open.

In alternative embodiments, the reasoner generally has more contextual information available to it in making the determination of whether the audio input includes a valid voice command. As introduced above, a history of previous utterances or possibly feedback from the assistants (e.g., from a dialog control component of an assistant indicating that it is expecting a follow-up utterance from the user) may add further evidence of whether an utterance is system directed and/or to which assistant it is directed. As another example, when there is only a driver in the car and no passengers, the reasoner can decide to initiate a dialog in a situation where otherwise the reasoner may have ignored an input because it appeared to be an inter-person communication. For example, recognition of “boy my Mercedes sure is comfortable” may be ignored when there are passengers, while prompting a “what can Mercedes do for you?” if the driver is alone in the car and therefore not communicating with anyone else. Similarly, if the reasoner has the context that the driver is on a telephone call (e.g., using the handsfree system), it may inhibit acting on any voice input, or have a higher threshold score in order to act.

As introduced above, the reasoner may be configured with words (e.g., names) that may appear in the position of a wake word in an utterance. For example, the names of users or other people in the acoustic environment may be known to the system. For example, “Angela,” “John,” “mom,” “dad” may be such words. These words may be configured in the system, or determined dynamically, for example, based on a personal key fob being used in a vehicle, personal cellphones being detected from their RF emissions etc. With this type of configuration, the reasoner can use occurrences of these words in the subject position of an utterance as an indication that the utterance is not a command intended for an assistant. Essentially, these words may be used as imposters for the names of assistants and/or used to replace placeholders in the grammar in which person names may occur. In the case of assistants whose names sound similar to such configured names of users or other people, performance of the system may be improved by reducing “false alarms” acting on utterances that are not intended to be commands.

As introduced above, the system may include or provide access to multiple different voice-based assistants. In some situations, an assistant may determine that it is not in fact the assistant that should process the command. In some embodiments, there is a mechanism to override the decision made by the reasoner, essentially rejecting the utterance followed by the reasoner passing the utterance on to another assistant. For example, based on an initial part of an utterance, a command may be classified as an in-vehicle command to be handled by a built-in vehicle based assistant, but one performing fuller NLU on the utterance, it does not map to a function of the vehicle-based assistant, and after being returned to the reasoner, the utterance may be sent instead to a home-based assistant.

Referring toFIG.2, an example of processing of an input audio signal begins with receiving the audio signal (step205). Note that in some embodiments, the receiving of the audio signal is triggered by a speech activity detector, or by an acoustic energy detector, so that the processing only occurs when there is a reasonable likelihood that the user is addressing the system. Other inputs, such as a camera input that uses facial characteristics (e.g., lip motion, facial expression, etc.), or an explicit “push-to-talk” button press, may be used to trigger when the acquiring or processing of the audio signal begins.

As discussed above, the audio signal is automatically transcribed (step210), while in parallel, the audio signal is passed to the reasoner. The text output from the transcribing (step21) and the audio signal are then processed to classify the input as system directed versus not system directed (step250). If the determination is that the input is not system directed (step251), the input is discarded (step290), while if the determination is that there was a command, the identified assistant is initiated (step220) and the command is interpreted and acted upon by the identified assistant (step230).

A variety of alternative implementations may be used to achieve similar user-visible functionality. The system above may be considered to be a two-class classifier: system-directed input vs extraneous input. The presence, lack of presence, and/or location of the WUW is one source of evidence provided to the reasoner, which essentially implements the classifier. The NLU output, as well as other contextual information constitutes other inputs to the reasoner implementing the classification.

In some alternative embodiments, the language model is more directly used in the classification. For example, language models statistics (e.g., word sequence probabilities, n-grams, etc.) are computed both for system-directed input as well as for extraneous input and are computed from separate training corpora. The reasoner can then compare the score of transcription under the two language models, for example, applying a threshold to a difference or ratio of the scores to classify the input as a system-directed input vs extraneous. Yet other approaches for “phrase spotting” approaches is implemented in the reasoner to detect utterances that conform exactly or statistically to target grammars or language models can similarly be used, without specifically spotting the WUW on its own.

In some alternatives, a difference or ratio of language model scores as introduced above, and/or phrase spotting scores for valid commands can be added as inputs to a reasoner that includes rule-based and/or procedural logic for making the classification that a system-directed command is part of the utterance.

In some examples, the reasoner implements a trained mapping of a number of inputs (e.g., including context), to make a classification of whether the input is a system-directed command vs extraneous input. For example, a machine-learning approach may be used for training, for example, an artificial neural network (ANN) or other classifier. In some such examples, the audio signal itself and/or the recognized text sequence, or other transformations of the input audio can also be provided to a trained classifier.

In some examples, a two-class classifier may be directly trained on the audio input (e.g., on a signal processed version of the audio input, such as a sequence of feature vectors) to classify the input as system-directed versus extraneous. In some such examples, the classifier essentially functions as a spotter or scorer of valid utterances that inherently weighs the presence of a WUW in an appropriate position within the utterance.

In some alternatives, the reasoner is not necessarily a separate component but rather is part of a dialog control. The dialog control component then uses the internally computed assessment (e.g., classification) of whether an utterance is a system-directed command in controlling the dialog. Furthermore, the dialog state informs the assessment, for example, increasing the belief that an utterance is a system-directed input if provided in response to a prompt from the system, for example, that is part of a confirmation or follow-up dialog.

As introduced above, the system as a whole can be configured, for example, with a user providing a customized name for an assistant, which in turn affects the WUW spotting and the logic implemented by the reasoner in determining whether an input is system directed. More generally, the reasoner may be configurable, for example, with each sub-assistant providing configuration information (e.g., rules, configuration data for a classifier, etc.) that the reasoner uses to determine if an input should be directed to that sub-assistant. Similarly, an NLU component may be configurable to provide a way of processing text input to get the meaning of the command that is to be passed to the assistant or sub-assistant.

Referring toFIG.3, in one implementation, the system is integrated into a vehicle310, with the microphone101, camera103, etc., monitoring the driver (not shown). The components illustrated inFIG.1are hosted on a computer312(i.e., an embedded computation system with a general purpose or special purpose processor), which may include non-transitory storage holding instructions to be executed by a processor of the computer to implement the functions described above. In some examples, the computer312connects to a user's smartphone314, which may host the assistant or one of the sub-assistants known to the system. In some examples, the smartphone314provides a communication link via a communication system320(e.g., via a cellular telephone system, e.g., a “5G” system) to a remote assistant330, such as a home-based assistant in the user's home remote from the vehicle. It should be understood that certain functions may be implemented in dedicated hardware rather than being performed by a processor, for example, with audio processing being performed by a hardware-based (i.e., dedicated circuitry) component. In some implementations, some or all of the functions that might be performed by the computer312are performed by a remote server computer in communication with components hosted in the vehicle.

A number of embodiments of the invention have been described. Nevertheless, it is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the following claims. Accordingly, other embodiments are also within the scope of the following claims. For example, various modifications may be made without departing from the scope of the invention. Additionally, some of the steps described above may be order independent, and thus can be performed in an order different from that described.