Patent Description:
A speech-enabled environment (e.g., home, workplace, school, automobile, etc.) allows a user to speak a query or a command out loud to a computer-based system that fields and answers the query and/or performs a function based on the command. The speech-enabled environment can be implemented using a network of connected microphone devices distributed through various rooms or areas of the environment. These devices may use hotwords to help discern when a given utterance is directed at the system, as opposed to an utterance that is directed to another individual present in the environment. Accordingly, the devices may operate in a sleep state or a hibernation state and wake-up only when a detected utterance includes a hotword. Neural networks have recently emerged as an attractive solution for training models to detect hotwords spoken by users in streaming audio. Typically, systems used to detect hotwords in streaming audio include a signal processing front end component, a neural network acoustic encoder component, and a hand-designed decoder component. These components are generally trained independent from one another, thereby creating added complexities and is suboptimal compared to training all components jointly,.

Document "<NPL>et al. discloses the modification of a trained deep neural network keyword detector by singular value decomposition (SVD) of the first network layer, motivated by the observation that the filters learned in the first network layer have significant structure and are thus amenable to compression.

The invention is defined by the independent claims <NUM> and <NUM>.

One aspect of the disclosure provides a method for detecting a hotword in streaming audio. The method includes, receiving, at data processing hardware of a user device, a sequence of input frames that each include respective audio features characterizing streaming audio captured by the user device, and generating, by the data processing hardware, a probability score indicating a presence of the hotword in the streaming audio using a memorized neural network. The memorized neural network includes sequentially-stacked single value decomposition filter (SVDF) layers, wherein each SVDF layer includes at least one neuron. Each neuron includes a respective memory component, a first stage, and a second stage. The respective memory component is associated with a respective memory capacity of the corresponding neuron. The first stage is configured to perform filtering on the respective audio features of each input frame individually and output the filtered audio features to the respective memory component. The second stage is configured to perform filtering on all the filtered audio features residing in the respective memory component. The method also includes determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold, and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio stream.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, a sum of the memory capacities associated with the respective memory components for a neuron from each of the SVDF layers provide the memorized neural network with a fixed memory capacity proportional to a length of time a typical speaker takes to speak the hotword. In some examples, the respective memory capacity associated with at least one of the respective memory components is different than the respective memory capacities associated with the remaining memory components. In other examples, the respective memory capacities associated with the respective memory components of all the SVDF layers is the same.

In some examples, a remote system trains the memorized neural network on a plurality of training input audio sequences. In these examples, each training input audio sequence includes: a sequence of input frames that each include one or more respective audio features characterizing phonetic components of the hotword; and labels assigned to the input frames, each label indicating a probability that the audio features of a respective input frame include a phonetic component of the hotword. In some configurations, training the memorized neural network includes, for each training input audio sequence: training an encoder portion by assigning a first label to a portion of the input frames that include a phonetic component of the hotword and assigning a second label to a remaining portion of the input frames that include phonetic components of the hotword; and training a decoder portion by applying a label indicating that the corresponding training input audio sequence either includes the hotword or does not include the hotword. Here, assigning the first label to the portion of the input audio frames may include: assigning the first label to at least one input frame that includes one or more respective audio features characterizing a last phonetic component of the hotword; and assigning the second label to the remaining input frames each including one or more respective audio features characterizing the remaining phonetic components of the hotword. In other configurations, training the memorized neural network includes, for each training input audio sequence: during a first stage of training, pre-training an encoder portion by assigning the labels to the input frames for the corresponding training input audio sequence; and during a second stage of training, initializing the encoder portion with the assigned labels from the first stage of training and training a decoder portion with outputs from the encoder portion to either detect the hotword or not detect the hotword.

The memorized neural network may include at least one additional processing layer disposed between adjacent SVDF layers. The memorized neural network includes at least one bottlenecking layer disposed between adjacent SVDF layers. In some examples the duio features of each input frame include log-filterbanks. For instance, each input frame may include forty log-filterbanks.

Another aspect of the disclosure provides a system for detecting audio in streaming audio. The system includes data processing hardware of a user device and memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a sequence of input frames that each include respective audio features characterizing streaming audio captured by the user device, and generating a probability score indicating a presence of the hotword in the streaming audio using a memorized neural network. The memorized neural network includes sequentially-stacked single value decomposition filter (SVDF) layers, wherein each SVDF layer includes at least one neuron. Each neuron includes a respective memory component, a first stage, and a second stage. The respective memory component is associated with a respective memory capacity of the corresponding neuron. The first stage is configured to perform filtering on the respective audio features of each input frame individually and output the filtered audio features to the respective memory component. The second stage is configured to perform filtering on all the filtered audio features residing in the respective memory component. The operations also include determining whether the probability score satisfies a hotword detection threshold, and when the probability score satisfies the hotword detection threshold, initiating a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio stream.

This aspect may include one or more of the following optional features. In some implementations, a sum of the memory capacities associated with the respective memory components for a neuron from each of the SVDF layers provide the memorized neural network with a fixed memory capacity proportional to a length of time a typical speaker takes to speak the hotword. In some examples, the respective memory capacity associated with at least one of the respective memory components is different than the respective memory capacities associated with the remaining memory components. In other examples, the respective memory capacities associated with the respective memory components of all the SVDF layers is the same.

A voice-enabled device (e.g., a user device executing a voice assistant) allows a user to speak a query or a command out loud and field and answer the query and/or perform a function based on the command. Through the use of a "hotword" (also referred to as a "keyword", "attention word", "wake-up phrase/word", "trigger phrase", or "voice action initiation command"), in which by agreement a predetermined term/phrase that is spoken to invoke attention for the voice enabled device is reserved, the voice enabled device is able to discern between utterances directed to the system (i.e., to initiate a wake-up process for processing one or more terms following the hotword in the utterance) and utterances directed to an individual in the environment. Typically, the voice-enabled device operates in a sleep state to conserve battery power and does not process input audio data unless the input audio data follows a spoken hotword. For instance, while in the sleep state, the voice-enabled device captures input audio via a microphone and uses a hotword detector trained to detect the presence of the hotword in the input audio. When the hotword is detected in the input audio, the voice-enabled device initiates a wake-up process for processing the hotword and/or any other terms in the input audio following the hotword.

Hotword detection is analogous to searching for a needle in a haystack because the hotword detector must continuously listen to streaming audio, and trigger correctly and instantly when the presence of the hotword is detected in the streaming audio. In other words, the hotword detector is tasked with ignoring streaming audio unless the presence of the hotword is detected. Neural networks are commonly employed by hotword detectors to address the complexity of detecting the presence of a hotword in a continuous stream of audio. A hotword detector typically includes three main components: a signal processing frontend; a neural network acoustic encoder; and a hand-designed decoder. The signal processing frontend may convert raw audio signals captured by the microphone of the user device into one or more audio features formatted for processing by the neural network acoustic encoder component. For instance, the neural network acoustic encoder component may convert these audio features into phonemes and the hand-designed decoder uses a hand-coded algorithm to stitch the phonemes together to provide a probability of whether or not an audio sequence includes the hotword. Typically, these three components are trained and/or manually designed independently from one another, thereby creating added complexity and loss in efficiency during training compared to training all the components jointly. Moreover, deploying models composed of independently trained models consume additional resource requirements (e.g., processing speeds and memory consumption). Separate models are often required for detecting different hotwords, as well as for detecting the same hotword in different locals. For example, an English speaker in South Africa may pronounce the phrase "Ok Google" differently than an English speaker in the United States that is located in North Dakota.

Implementations herein are directed toward an end-to-end hotword spotting system (also referred to as a 'keyword spotting system') that trains both encoding and decoding components into a single memorized neural network to determine a probability of a presence of a designated hotword in streaming audio. This single memorized neural network may be trained to detect multiple hotwords, as well as detecting a same hotword spoken in different languages and/or different locals. Specifically, the memorized neural network refers to a neural network topology having an amount of fixed memory proportional to an amount of streaming audio the neural network wants to remember into the past. For instance, it may be desirable for the neural network to have only enough memory to remember an amount of streaming audio equivalent to the time a typical speaker takes to speak a designated hotword. In some implementations, the memorized neural network topology is a layered topology of Single Value Decomposition Filter (SVDF) layers, with each layer including one or more SVDF neurons. Each SVDF neuron of each layer includes a respective memory capacity and the memory capacities of all of the SVDF layers additively make-up the total fixed memory for the neural network to remember only a fixed length of time in the streaming audio that is necessary to capture audio features characterizing the hotword. Each neuron may also include an appropriate activation function (e.g., rectified linear). Additionally, as the output of each SVDF layer is an input to a subsequent SVDF layer, bottleneck layers may be disposed between one or more adjacent SVDF layers to scale the number of inputs fed to subsequent SVDF layers.

Referring to <FIG>, in some implementations, an example system <NUM> includes one or more user devices <NUM> each associated with a respective user <NUM> and in communication with a remote system <NUM> via a network <NUM>. Each user device <NUM> may correspond to a computing device, such as a mobile phone, computer, wearable device, smart appliance, smart speaker, etc., and is equipped with data processing hardware <NUM> and memory hardware <NUM>. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware). The user device <NUM> receives a trained memorized neural network <NUM> from the remote system <NUM> via the network <NUM> and executes the trained memorized neural network <NUM> to detect hotwords in streaming audio <NUM>. The trained memorized neural network <NUM> may reside in a hotword detector <NUM> (also referred to as a hotworder) of the user device <NUM> that is configured to detect the presence of a hotword in streaming audio without performing semantic analysis or speech recognition processing on the streaming audio <NUM>. Optionally, the trained memorized neural network <NUM> may additionally or alternatively reside in an automatic speech recognizer (ASR) <NUM> of the user device <NUM> and/or the remote system <NUM> to confirm that the hotword detector <NUM> correctly detected the presence of a hotword in streaming audio <NUM>.

In some implementations, the data processing hardware <NUM> trains the memorized neural network <NUM> using training samples <NUM> obtained from annotated utterance pools <NUM>. The annotated utterance pools <NUM> may reside on the memory hardware <NUM> and/or some other remote memory location(s). In the example shown, when the user <NUM> speaks an utterance <NUM> including a hotword (e.g., "Hey Google") captured as streaming audio <NUM> by the user device <NUM>, the memorized neural network <NUM> executing on the user device <NUM> is configured to detect the presence of the hotword in the utterance <NUM> to initiate a wake-up process on the user device <NUM> for processing the hotword and/or one or more other terms (e.g., query or command) following the hotword in the utterance <NUM>. In additional implementations, the user device <NUM> sends the utterance <NUM> to the remote system <NUM> for additional processing or verification (e.g., with another, potentially more computationally-intensive memorized neural network <NUM>).

In the example shown, the memorized neural network <NUM> includes an encoder portion <NUM> and a decoder portion <NUM> each including a layered topology of single value decomposition filter (SVDF) layers <NUM>. The SVDF layers <NUM> provide the memory for the neural network <NUM> by providing each SVDF layer <NUM> with a memory capacity such that the memory capacities of all of the SVDF layers <NUM> additively make-up the total fixed memory for the neural network <NUM> to remember only a fixed length of time in the streaming audio <NUM> necessary to capture audio features <NUM> (<FIG> and <FIG>) that characterize the hotword.

Referring now to <FIG>, a typical hotword detector uses a neural network acoustic encoder <NUM> without memory. Because the network <NUM> lacks memory, each neuron <NUM> of the acoustic encoder <NUM> must accept, as an input, every audio feature of every frame <NUM>, 210a-d of a spoken utterance <NUM> simultaneously. Note that each frame <NUM> can have any number of audio features, each of which the neuron <NUM> accepts as an input. Such a configuration requires a neural network acoustic encoder <NUM> of substantial size that increases dramatically as the fixed length of time increases and/or the number of audio features increases. The output of the acoustic encoder <NUM> results in a probability of each, for example, phoneme of the hotword that has been detected. The acoustic encoder <NUM> must then rely on a hand-coded decoder to process the outputs of the acoustic encoder <NUM> (e.g., stitch together the phonemes) in order to generate a score (i.e., an estimation) indicating a presences of the hotword.

Referring now to <FIG> and <FIG>, according to the invention, a single value decomposition filter (SVDF) neural network <NUM> (also referred to as a memorized neural network) has any number of neurons/nodes <NUM>, where each neuron <NUM> accepts only a single frame <NUM>, 210a-d of a spoken utterance <NUM> at a time. That is, if each frame <NUM>, for example, constitutes <NUM> of audio data, a respective frame <NUM> is input to the neuron <NUM> approximately every <NUM> (i.e., Time <NUM>, Time <NUM>, Time <NUM>, Time <NUM>, etc.). <FIG> shows each neuron <NUM> including a two-stage filtering mechanism: a first stage <NUM> (i.e., Stage <NUM> Feature Filter) that performs filtering on a features dimension of the input and a second stage <NUM> (i.e., Stage <NUM> Time Filter) that performs filtering on a time dimension on the outputs of the first stage <NUM>. Therefore, the stage <NUM> feature filter <NUM> performs feature filtering on only the current frame <NUM>. The result of the processing is then placed in a memory component <NUM>. The size of the memory component <NUM> is configurable per node or per layer level. After the stage <NUM> feature filter <NUM> processes a given frame <NUM> (e.g., by filtering audio features within the frame), the filtered result is placed in a next available memory location <NUM>, 332a-d of the memory component <NUM>. Once all memory locations <NUM> are filled, the stage <NUM> feature filter <NUM> will overwrite the memory location <NUM> storing the oldest filtered data in the memory component <NUM>. Note that, for illustrative purposes, <FIG> shows a memory component <NUM> of size four (four memory locations 332a-d) and four frames 210a-d, but due to the nature of hotword detection, the system <NUM> will typically monitor streaming audio <NUM> continuously such that each neuron <NUM> will "slide" along or process frames <NUM> akin to a pipeline. Put another way, if each stage includes N feature filters <NUM> and N time filters <NUM> (each matching the size of the input feature frame <NUM>), the layer is analogous to computing N x T (T equaling the number of frames <NUM> in a fixed period of time) convolutions of the feature filters by sliding each of the N filters <NUM>, <NUM> on the input feature frames <NUM>, with a stride the size of the feature frames. For example, since the example shows the memory component <NUM> at capacity after the stage <NUM> feature filter outputs the filtered audio features associated with Frame <NUM> (F4) 210d (during Time <NUM>), the stage <NUM> feature filter <NUM> would place filtered audio features associated with following Frame <NUM> (F5) (during a Time <NUM>) into memory <NUM> by overwriting the filtered audio features associated with Frame <NUM> (F1) 210a within memory location 332a. In this way, the stage <NUM> time filter <NUM> applies filtering to the previous T - <NUM> (T again equaling the number of frames <NUM> in a fixed period of time) filtered audio features output from the stage <NUM> feature filter <NUM>.

The stage <NUM> time filter <NUM> then filters each filtered audio feature stored in memory <NUM>. For example, <FIG> shows the stage <NUM> time filter <NUM> filtering the audio features in each of the four memory locations <NUM> every time the stage <NUM> feature filter <NUM> stores a new filtered audio feature into memory <NUM>. In this way, the stage <NUM> time filter <NUM> is always filtering a number of past frames <NUM>, where the number is proportional to the size of the memory <NUM>. Each neuron <NUM> is part of a single SVDF layer <NUM>, and the neural network <NUM> may include any number of layers <NUM>. The output of each stage <NUM> time filter <NUM> is passed to an input of a neuron <NUM> in the next layer <NUM>. The number of layers <NUM> and the number of neurons <NUM> per layer <NUM> is fully configurable and is dependent upon available resources and desired size, power, and accuracy. This disclosure is not limited to the number of SVDF layers <NUM> nor the number of neurons <NUM> in each SVDF layer <NUM>.

Referring now to <FIG>, each SVDF layer <NUM>, 302a-n (or simply 'layer') of the neural network <NUM> is connected such that the outputs of the previous layer are accepted as inputs to the corresponding layer <NUM>. In some examples, the final layer 302n outputs a probability score <NUM> indicating the probability that the utterance <NUM> includes the hotword.

In an SVDF network <NUM>, the layer design derives from the concept that a densely connected layer <NUM> that is processing a sequence of input frames <NUM> can be approximated by using a singular value decomposition of each of its nodes <NUM>. The approximation is configurable. For example, a rank R approximation signifies extending a new dimension R for the layer's filters: stage <NUM> occurs independently, and in stage <NUM>, the outputs of all ranks get added up prior to passing through the non-linearity. In other words, an SVDF decomposition of the nodes <NUM> of a densely connected layer of matching dimensions can be used to initialize an SVDF layer <NUM>, which provides a principled initialization and increases the quality of the layer's generalization. In essence, the "power" of a larger densely connected layer is transferred into a potentially (depending on the rank) much smaller SVDF. Note, however, the SVDF layer <NUM> does not need the initialization to outperform a densely connected or even convolutional layer with the same or even more operations.

Thus, implementations herein are directed toward a stateful, stackable neural network <NUM> where each neuron <NUM> of each SVDF layer <NUM> includes a first stage <NUM>, associated with filtering audio features, and a second stage <NUM>, associated with filtering outputs of the first stage <NUM> with respect to time. Specifically, the first stage <NUM> is configured to perform filtering on one or more audio features on one audio feature input frame <NUM> at a time and output the filtered audio features to the respective memory component <NUM>. Here, the stage <NUM> feature filter <NUM> receives one or more audio features associated with a time frame <NUM> as input for processing and outputs the processed audio features into the respective memory component <NUM> of the SVDF layer <NUM>. Thereafter, the second stage <NUM> is configured to perform filtering on all the filtered audio features output from the first stage <NUM> and residing in the respective memory component <NUM>. For instance, when the respective memory component <NUM> is equal to eight (<NUM>), the second stage <NUM> would pull up to the last eight (<NUM>) filtered audio features residing in the memory component <NUM> that were output from the first stage <NUM> during individual filtering of the audio features within a sequence of eight (<NUM>) input frames <NUM>. As the first stage <NUM> fills the corresponding memory component <NUM> to capacity, the memory locations <NUM> containing the oldest filtered audio features are overwritten (i.e., first in, first out). Thus, depending on the capacity of the memory component <NUM> at the SVDF neuron <NUM> or layer <NUM>, the second stage <NUM> is capable of remembering a number of past outputs processed by the first stage <NUM> of the corresponding SVDF layer <NUM>. Moreover, since the memory components <NUM> at the SVDF layers <NUM> are additive, the memory component <NUM> at each SVDF neuron <NUM> and layer <NUM> also includes the memory of each preceding SVDF neuron <NUM> and layer <NUM>, thus extending the overall receptive field of the memorized neural network <NUM>. For instance, in a neural network <NUM> topology with four SVDF layers <NUM>, each having a single neuron <NUM> with a memory component <NUM> equal to eight (<NUM>), the last SVDF layer <NUM> will include a sequence of up to the last thirty-two (<NUM>) audio feature input frames <NUM> individually filtered by the neural network <NUM>. Note, however, the amount of memory is configurable per layer <NUM> or even per node <NUM>. For example, the first layer 302a may be allotted thirty-two (<NUM>) locations <NUM>, while the last layer <NUM> may be configured with eight (<NUM>) locations <NUM>. As a result, the stacked SVDF layers <NUM> allow the neural network <NUM> to process only the audio features for one input time frame <NUM> (e.g., <NUM> milliseconds of audio data) at a time and incorporate a number of filtered audio features into the past that capture the fixed length of time necessary to capture the designated hotword in the streaming audio <NUM>. By contrast, a neural network <NUM> without memory (as shown in <FIG>) would require its neurons <NUM> to process all of the audio feature frames covering the fixed length of time (e.g., <NUM> seconds of audio data) at once in order to determine the probability of the streaming audio including the presence of the hotword, which drastically increases the overall size of the network. Moreover, while recurrent neural networks (RNNs) using long short-term memory (LSTM) provide memory, RNN-LSTMs cause the neurons to continuously update their state after each processing instance, in effect having an infinite memory, and thereby prevent the ability to remember a finite past number of processed outputs where each new output re-writes over a previous output (once the fixed-sized memory is at capacity). Put another way, SVDF networks do not recur the outputs into the state (memory), nor rewrite all the state with each iteration; instead, the memory keeps each inference run's state isolated from subsequent runs, instead pushing and popping in new entries based on the memory size configured for the layer.

Referring now to <FIG> and <FIG>, in some implementations, the memorized neural network <NUM> is trained on a plurality of training input audio sequences <NUM> (i.e., training samples) that each include a sequence of input frames <NUM>, 210a-n and labels <NUM> assigned to the input frames <NUM>. Each input frame <NUM> includes one or more respective audio features <NUM> characterizing phonetic components <NUM> of a hotword, and each label <NUM> indicates a probability that the one or more audio features <NUM> of a respective input frame <NUM> include a phonetic component <NUM> of the hotword. In some examples, the audio features <NUM> for each input frame <NUM> are converted from raw audio signals <NUM> of an audio stream <NUM> during a pre-processing stage <NUM>. The audio features <NUM> may include one or more log-filterbanks. Thus, the pre-processing stage may segment the audio stream <NUM> (or spoken utterance <NUM>) into the sequence of input frames <NUM> (e.g., <NUM> each), and generate separate log-filterbanks for each frame <NUM>. For example, each frame <NUM> may be represented by forty log-filterbanks. Moreover, each successive SVDF layer <NUM> receives, as input, the filtered audio features <NUM> with respect to time that are output from the immediately preceding SVDF layer <NUM>.

In the example shown, each training input audio sequence <NUM> is associated with a training sample that includes an annotated utterance containing a designated hotword occurring within a fixed length of time (e.g., two seconds). The memorized neural network <NUM> may also optionally be trained on annotated utterances <NUM> that do not include the designated hotword, or include the designated hotword but spanning a time longer than the fixed length of time, and thus, would not be falsely detected due to the fixed memory forgetting data outside the fixed length of time. In some examples, the fixed length of time corresponds to an amount of time that a typical speaker would take to speak the designated hotword to summon a user device <NUM> for processing spoken queries and/or voice commands. For instance, if the designated hotword includes the phrase "Hey Google" or "Ok Google", a fixed length of time set equal to two seconds is likely sufficient since even a slow speaker would generally not take more than two seconds to speak the designated phrase. Accordingly, since it is only important to detect the occurrence of the designated hotword within streaming audio <NUM> during the fixed length of time, the neural network <NUM> includes an amount of fixed memory that is proportional to the amount of audio to span the fixed time (e.g., two seconds). Thus, the fixed memory of the neural network <NUM> allows neurons <NUM> of the neural network to filter audio features <NUM> (e.g., log-filterbanks) from one input frame <NUM> (e.g., <NUM> time window) of the streaming audio <NUM> at a time, while storing the most recent filtered audio features <NUM> spanning the fixed length of time and removing or deleting any filtered audio features <NUM> outside the fixed length of time from a current filtering iteration. Thus, if the neural network <NUM> has, for example, a memory depth of thirty-two (<NUM>), the first thirty-two (<NUM>) frames processed by the neural network <NUM> will fill the memory component <NUM> to capacity, and for each new output after the first <NUM>, the neural network <NUM> will remove the oldest processed audio feature from the corresponding memory location <NUM> of the memory component <NUM>.

Referring to <FIG>, for end-to-end training, training input audio sequence 400a includes labels <NUM> that may be applied to each input frame <NUM>. In some examples, when a training sample 400a contains the hotword, a target label <NUM> associated with a target score (e.g., '<NUM>') is applied to one or more input frames <NUM> that contain audio features <NUM> characterizing phonetic components <NUM> at or near the end of the hotword. For example, if the phonetic components <NUM> of the hotword "OK Google" are broken into: "ou", 'k', "eI", "<silence>", 'g', 'u', 'g', '@', 'l', then target labels of the number '<NUM>' are applied to all input frames <NUM> that correspond to the letter 'l' (i.e. the last component <NUM> of the hotword), which are part of the required sequence of phonetic components <NUM> of the hotword. In this scenario, all other input frames <NUM> (not associated with the last phonetic component <NUM>) are assigned a different label (e.g., '<NUM>'). Thus, each input frame <NUM> includes a corresponding input feature-label pair <NUM>, <NUM>. The input features <NUM> are typically one-dimensional tensors corresponding to, for example, mel filterbanks or log-filterbanks, computed from the input audio over the input frame <NUM>. The labels <NUM> are generated from the annotated utterances 400a, where each input feature tensor <NUM> is assigned a phonetic class via a force-alignment step (i.e., a label of '<NUM>' is given to pairs corresponding to the last class belonging to the hotword, and '<NUM>' to all the rest). Thus, the training input audio sequence 400a includes binary labels assigned to the sequence of input frames. The annotated utterances 400a, or training input audio sequence 400a, correspond to the training samples <NUM> obtained from the annotated utterance pools <NUM> of <FIG>.

In another implementation, <FIG> includes a training input audio sequence 400b that includes labels <NUM> associated with scores that increase along the sequence of input frames <NUM> as the number of audio features <NUM> characterizing (matching) phonetic components <NUM> of the hotword progresses. For instance, when the hotword includes "Ok Google", the input frames <NUM> that include respective audio features <NUM> that characterize the first phonetic components, 'o' and 'k', have assigned labels <NUM> of '<NUM>', while the input frames <NUM> that include respective audio features <NUM> characterizing the final phonetic component of 'l' have assigned labels <NUM> of '<NUM>'. The input frames <NUM> including respective audio features <NUM> characterizing the middle phonetic components <NUM> have assigned labels <NUM> of `<NUM>', '<NUM>', and '<NUM>'.

In additional implementations, the number of positive labels <NUM> increases. For example, a fixed amount of '<NUM>' labels <NUM> is generated, starting from the first frame <NUM> including audio features <NUM> characterizing to the final phonetic component <NUM> of the hotword. In this implementation, when the configured number of positive labels <NUM> (e.g., '<NUM>') is large, a positive label <NUM> may be applied to frames <NUM> that otherwise would have been applied a non-positive label <NUM> (e.g., '<NUM>'). In other examples, the start position of the positive label <NUM> is modified. For example, the label <NUM> may be shifted to start at either a start, mid-point, or end of a segment of frames <NUM> containing the final keyword phonetic component <NUM>. Still yet in other examples, a weight loss is associated with the input sequence. For example, weight loss data is added to the input sequence that allows the training procedure to reduce the loss (i.e. error gradient) caused by small mis-alignment. Specifically, with frame-based loss functions, a loss can be caused from either mis-classification or mis-alignment. To reduce the loss, the neural network <NUM> predicts both the correct label <NUM> and correct position (timing) of the label <NUM>. Even if the network <NUM> detected the keyword at some point, the result can be considered an error if it's not perfectly aligned with the given target label <NUM>. Thus, weighing the loss is particularly useful for frames <NUM> with high likelihood of mis-alignment during the force-alignment stage.

As a result of training using either of the training input audio sequences 400a, 400b of <FIG> and <FIG>, the neural network <NUM> is optimized (typically using cross-entropy (CE) loss) to output binary decision labels <NUM> indicating whether the hotword(s) are present in the streaming audio <NUM>. In some examples, network <NUM> is trained in two stages. Referring now to <FIG>, schematic view 500a shows an encoder portion (or simply 'encoder') 310a of the neural network <NUM> that includes, for example, eight layers, that are trained individually to produce acoustic posterior probabilities. In addition to the SVDF layers, the network <NUM> may, for example, include bottleneck, softmax, and/or other layers. For training the encoder <NUM>10a, label generation assigns distinct classes to all the phonetic components of the hotword (plus silence and "epsilon" targets for all that is not the hotword). Then, the decoder portion (or simply 'decoder') 311a of the neural network <NUM> is trained by creating a topology where the first part (i.e. the layers and connections) matches that of the encoder 310a, and a selected checkpoint from that encoder 310a of the neural network <NUM> is used to initialize it. The training is specified to "freeze" (i.e. not update) the parameters of the encoder 310a, thus tuning just the decoder 311a portion of the topology. This naturally produces a single spotter neural network, even though it is the product of two staggered training pipelines. Training with this method is particularly useful on models that tend to present overfitting to parts of the training set.

Alternatively, the neural network <NUM> is trained end-to-end from the start. For example, the neural network <NUM> accepts features directly (similarly to the encoder 310a training described previously), but instead uses the binary target label <NUM> (i.e., '<NUM>' or '<NUM>') outputs for use in training the decoder 311a. Such an end-to-end neural network <NUM> may use any topology. For example, as shown in <FIG>, schematic view 500b shows a neural network <NUM> topology of an encoder 310b and a decoder 311b that is similar to the topology of <FIG> except that the encoder 310b does not include the intermediate softmax layer. As with the topology of <FIG>, the topology of <FIG> may use a pre-trained encoder checkpoint with an adaptation rate to tune how the decoder 311b part is adjusted (e.g. if the adaptation rate is set to <NUM>, it is equivalent to the <FIG> topology). This end-to-end pipeline, where the entirety of the topology's parameters are adjusted, tends to outperform the separately trained encoder 310a and decoder 311a of <FIG>, particularly in smaller sized models which do not tend to overfit.

Thus, neural network <NUM> avoids the use of a manually tuned decoder. Manual tuning the decoder increases the difficulty in changing or adding hotwords. The single memorized neural network <NUM> can be trained to detect multiple different hotwords, as well as the same hotword across two or more locales. Further, detection quality reduces compared to a network optimized specifically for hotword detection trained with potentially millions of examples. Further, typical manually tuned decoders are more complicated than a single neural network that performs both encoding and decoding. Traditional systems tend to be overparameterized, consuming significantly more memory and computation than a comparable end-to-end model and they are unable to leverage as much neural network acceleration hardware. Additionally, a manual tuned decoder suffers from accented utterances, and makes it extremely difficult to create detectors that can work across multiple locales and/or languages.

The memorized neural network <NUM> outperforms simple fully-connected layers of the same size, but also benefits from optionally initializing parameters from a pre-trained fully connected layer. The network <NUM> allows fine grained control over how much to remember from the past. This results in outperforming RNN-LSTMs for certain tasks that do not benefit (and actually are hurt) from paying attention to theoretically infinite past (e.g. continuously listening to streaming audio). However, network <NUM> can work in tandem with RNN-LSTMs, typically leveraging SVDF for the lower layers, filtering the noisy low-level feature past, and LSTM for the higher layers. The number of parameters and computation are finely controlled, given that several relatively small filters comprise the SVDF. This is useful when selecting a tradeoff between quality and size/computation. Moreover, because of this quality, network <NUM> allows creating very small networks that outperform other topologies like simple convolutional neural networks (CNNs) which operate at a larger granularity.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of detecting a hotword in streaming audio <NUM>. The flowchart start at operation <NUM> by receiving, at data processing hardware <NUM> of a user device <NUM>, a sequence of input frames <NUM> that each include respective audio features <NUM> characterizing streaming audio <NUM> captured by the user device <NUM>. The audio features <NUM> of each input frame <NUM> may include log-filterbanks. For example, each input frame <NUM> may include forty log-filterbanks. At operation <NUM>, the method <NUM> includes generating, by the data processing hardware <NUM>, a probability score <NUM> indicating a presence of a hotword in the streaming audio <NUM> using a memorized neural network <NUM> including sequentially-stacked SVDF layers <NUM>, wherein each SVDF layer <NUM> includes at least one neuron <NUM>, and each neuron <NUM> includes a respective memory component <NUM>, the respective memory component <NUM> associated with a respective memory capacity of the corresponding neuron <NUM>. Each neuron <NUM> also includes a first stage <NUM> and a second stage <NUM>. The first stage <NUM> is configured to perform filtering on audio features <NUM> of each input frame <NUM> individually and output the filtered audio features <NUM> to the respective memory component <NUM>. The second stage <NUM> is configured to perform filtering on all the filtered audio features <NUM> residing in the respective memory component <NUM>. The neural network <NUM> may include at least one additional processing layer disposed between adjacent SVDF layers <NUM>. The neural network <NUM>, in some examples, includes at least one bottlenecking layer disposed between adjacent SVDF layers <NUM>. Bottleneck layers are used to significantly reduce the parameter count between layers.

In some examples, a sum of the memory capacities associated with the respective memory components <NUM> for a neuron <NUM> from each of the SVDF layers <NUM> provide the neural network <NUM> with a fixed memory capacity proportional to a length of time a typical speaker takes to speak the hotword. The respective memory capacity associated with at least one of the respective memory components <NUM> may be different than the respective memory capacities associated with the remaining memory components <NUM>. Alternatively, the respective memory capacities associated with the respective memory components <NUM> of the neurons <NUM> of all the SVDF layers <NUM> is the same.

At operation <NUM>, the method <NUM> includes determining, by the data processing hardware <NUM>, whether the probability score <NUM> satisfies a hotword detection threshold. When the probability score <NUM> satisfies the hotword detection threshold, the method <NUM> includes, at operation <NUM>, initiating, by the data processing hardware <NUM>, a wake-up process on the user device <NUM> for processing the hotword and/or one or more other terms following the hotword in the audio stream <NUM>.

In some implementations, a remote system <NUM> having computing resources <NUM> and memory resources <NUM> is configured to train the neural network <NUM> on a plurality of training input sequences <NUM>, each training input audio sequence <NUM> including a sequence of input frames <NUM> that each include one or more respective audio features <NUM> characterizing phonetic components <NUM> of the hotword. Each training input audio sequence <NUM> also includes labels <NUM> assigned to the input frames <NUM>, each label <NUM> indicating a probability that the audio features <NUM> of a respective input frame <NUM> include a phonetic component <NUM> of the hotword. In additional examples, training the neural network <NUM> includes, for each training input audio sequence <NUM>, training an encoder portion 310b by assigning a first label <NUM> to a portion of the input frames <NUM> that include a phonetic component <NUM> of the hotword. The training also includes assigning a second label <NUM> to a remaining portion of the input frames <NUM> that includes phonetic components <NUM> of the hotword and training a decoder portion 311b by applying a label <NUM> indicating that the corresponding training input audio sequence <NUM> either includes the hotword or does not include the hotword. Assigning the first label <NUM> to the portion of the input frames <NUM> may include assigning the first label <NUM> to at least one input frame <NUM> that includes one or more respective audio features <NUM> characterizing a last phonetic component <NUM> of the hotword and assigning the second labels <NUM> to the remaining input frames <NUM> each including one or more respective audio features <NUM> characterizing the remaining phonetic components of the hotword.

In some implementations, the method <NUM> includes training the neural network <NUM> by, during a first stage <NUM> of training, pre-training an encoder portion 310a by assigning the labels <NUM> to the input frames <NUM> for the corresponding training input audio sequence <NUM>. During a second stage <NUM> of training, the method <NUM> includes initializing the encoder portion <NUM>10a with the assigned labels <NUM> from the first stage of training and training a decoder portion 311a with outputs from the encoder portion <NUM> to either detect the hotword or not detect the hotword.

The computing device <NUM> includes a processor <NUM>, memory <NUM>, a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus770 and a storage device <NUM>.

The memory <NUM> stores information non-transitorily within the computing device 370a.

For example, it may be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>) of a user device (<NUM>), a sequence of input frames (<NUM>) that each include respective audio features (<NUM>) characterizing streaming audio (<NUM>) captured by the user device (<NUM>);
generating, by the data processing hardware (<NUM>), a probability score (<NUM>) indicating a presence of a hotword in the streaming audio (<NUM>) using a memorized neural network (<NUM>), the memorized neural network (<NUM>) comprising sequentially-stacked single value decomposition filter (SVDF) layers (<NUM>), wherein each SVDF layer (<NUM>) comprises at least one neuron (<NUM>), and each neuron (<NUM>) comprises:
a respective memory component (<NUM>), the respective memory component (<NUM>) associated with a respective memory capacity of the corresponding neuron (<NUM>) and comprising a plurality of memory locations (<NUM>);
a first stage (<NUM>) configured to perform filtering on the respective audio features (<NUM>) of each input frame (<NUM>) individually and output the filtered audio features (<NUM>) to the respective memory component (<NUM>); and
a second stage (<NUM>) configured to perform filtering on all the filtered audio features (<NUM>) residing in the respective memory component (<NUM>);
wherein each neuron is configured such that:
the first stage performs filtering on the respective audio features (<NUM>) of each input frame (<NUM>) individually and outputs the filtered audio features (<NUM>) to a next available memory location (<NUM>) of the respective memory component (<NUM>), wherein once all of the memory locations (<NUM>) of the respective memory component (<NUM>) are filled, the first stage overwrites the memory location storing the oldest filtered audio features (<NUM>) in the respective memory component (<NUM>), and then
the second stage performs filtering on all of the filtered audio features residing in the respective memory component (<NUM>);
determining, by the data processing hardware (<NUM>), whether the probability score (<NUM>) satisfies a hotword detection threshold; and
when the probability score (<NUM>) satisfies the hotword detection threshold, initiating, by the data processing hardware (<NUM>), a wake-up process on the user device (<NUM>) for processing the hotword and/or one or more other terms following the hotword in the audio stream (<NUM>).