Patent Description:
A speech-enabled environment (e.g., home, workplace, school, automobile, etc.) allows a user to speak a query or a command out loud to a computer-based system that fields and answers the query and/or performs a function based on the command. The speech-enabled environment can be implemented using a network of connected microphone devices distributed through various rooms or areas of the environment. These devices may use hotwords to help discern when a given utterance is directed at the system, as opposed to an utterance that is directed to another individual present in the environment. Accordingly, the devices may operate in a sleep state or a hibernation state and wake-up only when a detected utterance includes a hotword. These devices may include two or more microphones to record multi-channel audio. Neural networks have recently emerged as an attractive solution for training models to detect hotwords spoken by users in streaming audio. Typically, neural networks used to detect hotwords in streaming audio receive a single channel of streaming audio.

"END-TO-END STREAMING KEYWORD SPOTTING" by Raziel Alvarez and Hyun-Jin Park describes an end-to-end keyword spotting system that, by subsuming both the encoding and decoding components into a single neural network, can be trained to produce directly an estimation (i.e. score) of the presence of a keyword in streaming audio. The system makes use of a type of neural network layer topology called SVDF (singular value decomposition filter).

One aspect of the disclosure provides a method for training a memorized neural network and using the trained memorized neural network to detect a hotword in a spoken utterance. The method includes receiving, at data processing hardware of a user device, a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware. Each channel of the streaming multi-channel audio includes respective audio features captured by a separate dedicated microphone in the array of microphones. For each input frame, the method includes processing, by the data processing hardware, using a three-dimensional (3D) singular value decomposition filter (SVDF) input layer of a memorized neural network, the respective audio features of each channel of the streaming multi-channel audio in parallel and generating, by the data processing hardware, using an intermediate layer of the memorized neural network, a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio. The method also includes generating, by the data processing hardware, using sequentially-stacked SVDF layers of the memorized neural network, a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame. The method also includes determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold. When the probability score satisfies the hotword detection threshold, the method includes initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the 3D SVDF input layer includes multiple parallel SVDF processing cells. Each SVDF processing cell of the multiple parallel SVDF processing cells is associated with a respective channel of the streaming multi-channel audio and configured to perform the processing on the respective audio features of the respective channel. In some examples, each SVDF processing cell includes at least one neuron, and each neuron includes a respective memory component, a first stage configured to perform filtering on the respective audio features of the respective channel of each input frame individually and output the filtered features to the respective memory component, and a second stage configured to perform filtering on all the filtered audio features residing in the respective memory component. The respective memory component is associated with a respective memory capacity of the corresponding neuron.

Optionally, the respective audio features of each respective channel of each input frame includes log-filterbanks. Each input frame may include forty log-filterbanks. The sequentially-stacked SVDF layers of the memorized neural network, in some examples, include an initial SVDF layer configured to receive the corresponding multi-channel audio feature representation of each input frame in sequence.

In some implementations, each sequentially-stacked SVDF layer includes at least one neuron, and each neuron includes a respective memory component, a first stage configured to perform filtering on the corresponding multi-channel audio feature representation of each input frame individually and output the filtered multi-channel audio feature representation to the respective memory component, and a second stage configured to perform filtering on all the filtered multi-channel audio feature representations residing in the respective memory component. The respective memory component is associated with a respective memory capacity of the corresponding neuron.

A sum of the memory capacities associated with the respective memory components for a neuron from each of the sequentially-stacked SVDF layers may provide the memorized neural network with a fixed memory capacity proportional to a length of time a typical speaker takes to speak the hotword. The respective memory capacity associated with at least one of the respective memory components may be different than the respective memory capacities associated with the remaining memory components. In some examples, the respective memory capacities associated with the respective memory components of all the sequentially-stacked SVDF layers is the same.

In some implementations, a remote system trains the memorized neural network on a plurality of multi-channel training input audio sequences. Each channel of each multi-channel training input audio sequence includes a sequence of respective input frames that each include one or more respective audio features characterizing phonetic components of the hotword and labels assigned to the respective input frames. Each label indicates a probability that the audio features of a respective input frame include a phonetic component of the hotword. In some examples, each channel of each corresponding multi-channel training input audio sequence among a first portion of the plurality of multi-channel training input audio sequences is a duplicate with each other channel of the corresponding multi-channel training input audio sequence and each channel of each corresponding multi-channel training input audio sequence among a remaining second portion of the plurality of multi-channel training input audio sequences is unique to each other channel of the corresponding multi-channel training input audio sequence. Optionally, the 3D SVDF input layer includes multiple parallel SVDF processing cells. Each SVDF processing cell of the multiple parallel SVDF processing cells is associated with a respective channel of each multi-channel training input audio sequence and configured to receive the respective audio features of each respective input frame of the respective channel individually.

Another aspect of the disclosure provides a system for training a memorized neural network and using the trained memorized neural network to detect a hotword in a spoken utterance. The system includes data processing hardware of a user device and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware. Each channel of the streaming multi-channel audio includes respective audio features captured by a separate dedicated microphone in the array of microphones. For each input frame, the operations include processing, using a three-dimensional (3D) single value decomposition filter (SVDF) input layer of a memorized neural network, the respective audio features of each channel of the streaming multi-channel audio in parallel and generating, using an intermediate layer of the memorized neural network, a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio. The operations also include generating, using sequentially-stacked SVDF layers of the memorized neural network, a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame. The operations also include determining whether the probability score satisfies a hotword detection threshold. When the probability score satisfies the hotword detection threshold, the operations include initiating a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio.

This aspect may include one or more of the following optional features. In some implementations, the 3D SVDF input layer includes multiple parallel SVDF processing cells. Each SVDF processing cell of the multiple parallel SVDF processing cells is associated with a respective channel of the streaming multi-channel audio and configured to perform the processing on the respective audio features of the respective channel. In some examples, each SVDF processing cell includes at least one neuron, and each neuron includes a respective memory component, a first stage configured to perform filtering on the respective audio features of the respective channel of each input frame individually and output the filtered features to the respective memory component, and a second stage configured to perform filtering on all the filtered audio features residing in the respective memory component. The respective memory component is associated with a respective memory capacity of the corresponding neuron.

A voice-enabled device (e.g., a user device executing a voice assistant) allows a user to speak a query or a command out loud and field and answer the query and/or perform a function based on the command. Through the use of a "hotword" (also referred to as a "keyword", "attention word", "wake-up phrase/word", "trigger phrase", or "voice action initiation command"), in which by agreement a predetermined term/phrase that is spoken to invoke attention for the voice enabled device is reserved, the voice enabled device is able to discern between utterances directed to the system (i.e., to initiate a wake-up process for processing one or more terms following the hotword in the utterance) and utterances directed to an individual in the environment. Typically, the voice-enabled device operates in a sleep state to conserve battery power and does not process input audio data unless the input audio data follows a spoken hotword. For instance, while in the sleep state, the voice-enabled device captures input audio via a microphone and uses a hotword detector trained to detect the presence of the hotword in the input audio. When the hotword is detected in the input audio, the voice-enabled device initiates a wake-up process for processing the hotword and/or any other terms in the input audio following the hotword.

Hotword detection is analogous to searching for a needle in a haystack because the hotword detector must continuously listen to streaming audio, and trigger correctly and instantly when the presence of the hotword is detected in the streaming audio. In other words, the hotword detector is tasked with ignoring streaming audio unless the presence of the hotword is detected. Neural networks are commonly employed by hotword detectors to address the complexity of detecting the presence of a hotword in a continuous stream of audio.

A hotword detector typically receives a single channel of audio recorded by a single microphone (i.e., mono audio) and determines presence of the hotword within the single channel of audio. Some user devices may include two or more microphones to record multi-channel streaming audio (i.e., one channel per microphone). In this case, the hotword detector typically will include a neural network for each channel, with each neural network processing a separate channel of audio to determine a presence of the hotword within the respective channel. The output of each neural network (i.e., the determination of the presence of the hotword within the respective channel) may be combined via logical OR. That is, if any of the neural networks determine presence of the hotword in their respective channel of streaming audio, the wake-up process for the user device is initiated. This technique drastically increases the computing resources (e.g., processing speed and memory consumption) necessary for the hotword detector. For example, a hotword detector that uses two models to process two channels of audio captured by two independent microphones may double the computing resources required over a single model processing a single channel of audio. Moreover, because each model processes only a single channel of audio, the hotword detector fails to take advantage potential benefits from using a microphone array to enhance noise robustness.

Implementations herein are directed toward an end-to-end multi-channel hotword spotting system (also referred to as a 'keyword spotting system') that trains a single memorized neural network to determine a probability of a presence of a designated hotword in multi-channel streaming audio. This single memorized neural network may be trained to detect multiple hotwords, as well as detecting a same hotword spoken in different languages and/or different locals. Specifically, the memorized neural network refers to a neural network topology having an amount of fixed memory proportional to an amount of multi-channel streaming audio the neural network wants to remember into the past For instance, it may be desirable for the neural network to have only enough memory to remember an amount of multi-channel streaming audio equivalent to the time a typical speaker takes to speak a designated hotword. In some implementations, the memorized neural network topology is a layered topology of that includes one or more three dimensional (3D) Singular Value Decomposition Filter (SVDF) layers, with each layer including two or more parallel SVDF processing cells. Each SVDF processing cell processes a separate channel of the multi-channel streaming audio simultaneously and in parallel with the other SVDF processing cells.

Each SVDF processing cell is configured to perform processing on respective audio features of the respective channel and includes at least one neuron having a respective memory capacity. Each neuron may also include an appropriate activation function (e.g., rectified linear). Additionally, the output of each SVDF processing cell is concatenated together and passed to a subsequent intermediate layer to generate a multi-channel audio feature representation of the streaming multi-channel audio.

Referring to <FIG>, in some implementations, an example system <NUM> includes one or more user devices <NUM> each associated with a respective user <NUM> and in communication with a remote system <NUM> via a network <NUM>. Each user device <NUM> may correspond to a computing device, such as a mobile phone, computer, wearable device, smart appliance, audio infotainment system, smart speaker, etc., and is equipped with data processing hardware <NUM> and memory hardware <NUM>. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware). The user device <NUM> receives a trained memorized neural network <NUM> from the remote system <NUM> via the network <NUM> and executes the trained memorized neural network <NUM> to detect hotwords in multi-channel streaming audio <NUM>. The multi-channel streaming audio <NUM> includes two or more channels <NUM>, 119a-n of audio. The trained memorized neural network <NUM> may reside in a multi-channel hotword detector <NUM> (also referred to as a hotworder) of the user device <NUM> that is configured to detect the presence of a hotword in streaming multi-channel audio without performing semantic analysis or speech recognition processing on the streaming multi-channel audio <NUM>. Optionally, the trained memorized neural network <NUM> may additionally or alternatively reside in an automatic speech recognizer (ASR) <NUM> of the user device <NUM> and/or the remote system <NUM> to confirm that the multi-channel hotword detector <NUM> correctly detected the presence of a hotword in the multi-channel streaming audio <NUM>.

In some implementations, the data processing hardware <NUM> trains the memorized neural network <NUM> using training samples <NUM> obtained from annotated utterance pools <NUM>. The annotated utterance pools <NUM> may reside on the memory hardware <NUM> and/or some other remote memory location(s). In the example shown, when the user <NUM> speaks an utterance <NUM> including a hotword (e.g., "Hey Google") captured as multi-channel streaming audio <NUM> by the user device <NUM>, the memorized neural network <NUM> executing on the user device <NUM> is configured to detect the presence of the hotword in the utterance <NUM> to initiate a wake-up process on the user device <NUM> for processing the hotword and/or one or more other terms (e.g., query or command) following the hotword in the utterance <NUM>. In additional implementations, the user device <NUM> sends the utterance <NUM> to the remote system <NUM> for additional processing or verification (e.g., with another, potentially more computationally-intensive memorized neural network <NUM>).

The user device may include (or be in communication with) two or more microphones <NUM>, 107a-n to capture the utterance <NUM> from the user <NUM>. Each microphone <NUM> may separately record the utterance <NUM> on a separate dedicated channel <NUM> of the multi-channel streaming audio <NUM>. For example, the user device <NUM> may include two microphones <NUM> that each record the utterance <NUM>, and the recordings from the two microphones <NUM> may be combined into two-channel streaming audio <NUM> (i.e., stereophonic audio or stereo). In some examples, the user device <NUM> may include more than two microphones. That is, the two microphones reside on the user device <NUM>. Additionally or alternatively, the user device <NUM> may be in communication with two or more microphones separate/remote from the user device <NUM>. For example, the user device <NUM> may be a mobile device disposed within a vehicle and in wired or wireless communication (e.g., Bluetooth) with two or more microphones of the vehicle. In some configurations, the user device <NUM> is in communication with least one microphone <NUM> residing on a separate device <NUM>, which may include, without limitation, an in-vehicle audio system, a computing device, a speaker, or another user device. In these configurations, the user device <NUM> may also be in communication with one or more microphones residing on the user device <NUM>.

In the example shown, the memorized neural network <NUM> includes an input three dimensional (3D) singular value decomposition filter (SVDF) layer <NUM> and a plurality of other layers <NUM>, e.g., sequentially-stacked SVDF layers <NUM>. The input 3D SVDF layer <NUM> processes audio features of each channel <NUM> of the streaming multi-channel audio <NUM> in parallel. That is, each channel <NUM> of the multi-channel streaming audio <NUM> is provided as input 3D SVDF layer <NUM> to process simultaneously.

Referring now to <FIG>, a typical hotword detection systems <NUM> may utilize multi-channel streaming audio <NUM> (i.e., audio captured by microphones 107a, 107b) by implementing a separate and independent end-to-end hotword detection model <NUM>, 202a-b for each channel <NUM>. Each model <NUM> receives a single channel 119a-b of streaming audio <NUM> and determines presence of a hotword within the respective channel <NUM>. The hotword determination for each model <NUM> is provided to a logical OR gate <NUM> to generate the hotword detection <NUM> (e.g., yes/no hotword detection or a probability score). That is, when any of the models <NUM> determine that the hotword is present within its respective channel <NUM> of streaming audio <NUM>, the system <NUM> initiates the wake up procedure of the user device <NUM>. With the system <NUM>, not only does each model <NUM> consume significant computing resources, none of the models <NUM> gain any potential benefit (e.g., noise robustness) present in a microphone array.

In contrast to the typical system <NUM> of <FIG> that implements independent models <NUM> for detecting the hotword on each channel <NUM> of multi-channel streaming audio <NUM>, <FIG> shows the multi-channel hotword detector <NUM> of the system <NUM> of <FIG> that uses a multi-channel end-to-end model (e.g., the memorized neural network <NUM>) configured to simultaneously receive both channels from the multi-channel streaming audio <NUM>. Here, not only are the computing resources approximately halved relative to the system <NUM> (i.e., one end-to-end model instead of two), the multi-channel model <NUM> may take advantage of the accuracy and noise robustness increases present when processing multi-channel streaming audio <NUM>. For example, the multi-channel model <NUM> may not only take advantage of redundancy in frequency features from each channel, but also exploit temporal variations across channels to enhance noise robustness. While the example shown depicts the multi-channel streaming audio <NUM> having only two channels and the end-to-end model <NUM> receiving each of the two channels 119a, 119b simultaneously, the model <NUM> can similarly receive three or more channels <NUM> simultaneously when the multi-channel streaming audio <NUM> includes three or more channels. Moreover, the model <NUM> may also be configured to receive only a single channel <NUM> when the streaming audio <NUM> includes only a single channel.

Referring now to <FIG>, in some implementations, the 3D SVDF neural network <NUM> (also referred to as a memorized neural network) includes at least one 3D SVDF layer <NUM>. Each 3D SVDF layer <NUM> includes two or more SVDF processing cells <NUM>, 304a-b and each SVDF processing cell <NUM> receives a sequence of input frames <NUM>, 210aa-be that characterize the streaming multi-channel audio <NUM> captured by the microphones <NUM>. Each SVDF processing cell <NUM> receives a separate channel of input frames 210a, 210b. In the example shown, the SVDF processing cell 304a receives input frames 210a that include audio features from a first channel 119a (i.e., channel <NUM>) captured by a first microphone 107a, while the SVDF processing cell 304b receives input frames 210b that include audio features from a second channel 119b (i.e., channel <NUM>) captured by a second microphone 107b. That is, the 3D SVDF input layer <NUM> may include multiple parallel SVDF processing cells <NUM>, and each SVDF processing cell <NUM> of the multiple parallel SVDF processing cells is associated with a respective channel <NUM> of the streaming multi-channel audio <NUM> and configured to perform processing on respective audio features <NUM> (<FIG>) of the respective channel <NUM>.

Each SVDF processing cell <NUM> has any number of neurons/nodes <NUM>, where each neuron <NUM> accepts only a single frame <NUM> of a spoken utterance <NUM> at a time. That is, if each frame <NUM>, for example, constitutes <NUM> of audio data, a respective frame <NUM> is input to the neuron <NUM> approximately every <NUM>. Each neuron <NUM> may include a two-stage filtering mechanism: a first stage <NUM> (i.e., αch0 and αch1) that performs filtering on a features dimension of the input and a second stage <NUM> (i.e., βt0 and βt1) that performs filtering on a time dimension on the outputs of the first stage <NUM>. Therefore, the stage <NUM> feature filter <NUM> performs feature filtering on only the current frame <NUM>. A result <NUM> of the processing is then placed in a memory component <NUM>. The size of the memory component <NUM> is configurable per node or per layer level. The respective memory capacity associated with at least one of the respective memory components <NUM> may be different than the respective memory capacities associated with the remaining memory components <NUM>. Alternatively, the respective memory capacities associated with the respective memory components <NUM> of the neurons <NUM> of all the SVDF processing cells <NUM> is the same.

After the stage <NUM> feature filter <NUM> processes a given frame <NUM> (e.g., by filtering audio features of the respective channel within the frame <NUM>), the filtered result <NUM> is placed in a next available memory location <NUM>, 332a-d of the memory component <NUM>. Once all memory locations <NUM> are filled, the stage <NUM> feature filter <NUM> will overwrite the memory location <NUM> storing the oldest filtered data in the memory component <NUM>. Note that, for illustrative purposes, <FIG> shows a memory component <NUM> of size four (four memory locations 332a-d) and five frames 210aa-e, 210ba-be, but due to the nature of hotword detection, the system <NUM> will typically monitor multi-channel streaming audio <NUM> continuously such that each neuron <NUM> will "slide" along or process frames <NUM> akin to a pipeline. Put another way, if each stage includes N feature filters <NUM> and N time filters <NUM> (each matching the size of the input feature frame <NUM>), the layer is analogous to computing N x T (T equaling the number of frames <NUM> in a fixed period of time) convolutions of the feature filters by sliding each of the N filters <NUM>, <NUM> on the input feature frames <NUM>, with a stride the size of the feature frames. For example, after the memory component <NUM> is at capacity after the stage <NUM> feature filter outputs the filtered audio features <NUM> into memory location 332d, the stage <NUM> feature filter <NUM> would place filtered audio features <NUM> associated with following frame (i.e., 210ae, 210be) into memory <NUM> by overwriting the filtered audio features <NUM> associated with frame 210aa, 210ba within memory location 332a. In this way, the stage <NUM> time filter <NUM> applies filtering to the previous T - <NUM> (T again equaling the number of frames <NUM> in a fixed period of time) filtered audio features output from the stage <NUM> feature filter <NUM>.

The stage <NUM> time filter <NUM> then filters each filtered audio feature stored in memory <NUM>. For example, <FIG> shows the stage <NUM> time filter <NUM> filtering the audio features in each of the four memory locations <NUM> every time the stage <NUM> feature filter <NUM> stores a new filtered audio feature into memory <NUM>. In this way, the stage <NUM> time filter <NUM> is always filtering a number of past frames <NUM>, where the number is proportional to the size of the memory <NUM>. Each neuron <NUM> is part of a single SVDF processing cell <NUM>, and the neural network <NUM> may include any number of processing cells <NUM>.

An output <NUM>, 342a-b of each stage <NUM> time filter <NUM> within a 3D SVDF layer <NUM> may be concatenated together to form a single output <NUM>. In the example shown, the output 342a of the SVDF processing cell 304a (i.e., O<NUM>) is concatenated with the output 342b of the SVDF processing cell 304b (i.e., O<NUM>) to form the single output <NUM>. The concatenated output <NUM> is passed as an input to a subsequent layer of the memorized neural network <NUM>. In some examples, the next layer may be another 3D SVDF layer <NUM> and the concatenated output <NUM> is passed to a neuron <NUM> of the next 3D SVDF layer <NUM>. In other examples, the subsequent layer is a fully-connected dense layer (<FIG>). The number of layers and the number of neurons <NUM> per layer is fully configurable and is dependent upon available resources and desired size, power, and accuracy. This disclosure is not limited to the number of 3D SVDF layers <NUM> (or other layer types), the number of SVDF processing cells <NUM> per 3D SVDF layer <NUM>, nor the number of neurons <NUM> in each SVDF processing cell <NUM>.

Referring now to <FIG>, the input 3D SVDF layer <NUM> of the neural network <NUM> is connected such that the output <NUM> is accepted as an input to a subsequent layer. In some implementations, the subsequent intermediate layer is a dense layer <NUM>. The dense layer <NUM> may generate a corresponding multi-channel audio feature representation <NUM> based on the concatenated output <NUM> received from the 3D SVDF layer(s) <NUM>. The dense layer is a fully-connected layer (i.e., every input is connected to every output) that processes the concatenated output <NUM>. In some examples, the dense layer <NUM> compensates for phase differences between the channels <NUM> of the multi-channel streaming audio <NUM>. That is, based on a difference in distance between the user <NUM> and each microphone <NUM>, each channel <NUM> has a different phase. Because the SVDF processing cells <NUM> include time filtering (i.e., the stage <NUM> time filter <NUM>), the dense layer <NUM> may process the concatenated output <NUM> to compensate or adjust for this phase delay similar to techniques used in beamforming.

In some implementations, subsequent layers include one or more additional 3D SVDF layers <NUM>. Subsequent layers include one or more SVDF layers <NUM> (e.g., sequentially-stacked SVDF layers <NUM>). The sequentially-stacked SVDF layers <NUM> may generate a probability score <NUM> indicating a presence of a hotword in the streaming multi-channel audio <NUM> based on the corresponding multi-channel audio feature representation <NUM> of each input frame <NUM>. The sequentially-stacked SVDF layers <NUM> include an initial SVDF layer 350a configured to receive the corresponding multi-channel audio feature representation <NUM>. Each SVDF layer <NUM> includes substantially the same components as each SVDF processing cell <NUM> of the 3D SVDF layer <NUM>. That is, each SVDF layer <NUM> and each SVDF processing cell <NUM> of the 3D SVDF layer <NUM> include at least one neuron <NUM> that includes the respective memory component <NUM>, stage <NUM> feature filter <NUM>, and stage <NUM> time filter <NUM>. The sequentially-stacked SVDF layers <NUM> may be referred to as two-dimensional (2D) SVDF layers, and similarly, as each SVDF processing cell <NUM> is associated with a respective channel of the multi-channel streaming audio <NUM>, each SVDF processing cell <NUM> also corresponds to a 2D SVDF layer that processes the respective audio features of the channel in parallel with the other SVDF processing cells <NUM> for each input frame <NUM> of the streaming multi-channel audio <NUM>. The parallel SVDF processing cells <NUM> of the 3D SVDF layer <NUM> add a third dimension (i.e., channels) to the two dimensions of the SVDF layers <NUM> (i.e., frequency and time). In some examples, the final layer 350n of the memorized neural network <NUM> outputs a probability score <NUM> indicating the probability that the utterance <NUM> includes the hotword. The system <NUM> may determine that the utterance <NUM> includes the hotword when the probability score satisfies a hotword detection threshold and initiate a wake-up process on the user device <NUM>.

Thus, implementations herein are directed toward a stateful, stackable neural network <NUM> that detects a hotword within a multi-channel stream of audio using three dimensions (i.e., time, frequency, and channel). A 3D SVDF layer <NUM> includes multiple SVDF processing cells <NUM> parallel. Each neuron <NUM> of each SVDF processing cell <NUM> includes a first stage <NUM>, associated with filtering audio features, and a second stage <NUM>, associated with filtering outputs of the first stage <NUM> with respect to time. Specifically, the first stage <NUM> is configured to perform filtering on one or more audio features on one audio feature input frame <NUM> at a time and output the filtered audio features to the respective memory component <NUM>. Here, the stage <NUM> feature filter <NUM> receives one or more audio features associated with a time frame <NUM> as input for processing and outputs the processed audio features into the respective memory component <NUM> of the SVDF processing cell <NUM>. Thereafter, the second stage <NUM> is configured to perform filtering on all the filtered audio features output from the first stage <NUM> and residing in the respective memory component <NUM>. For instance, when the respective memory component <NUM> is equal to eight (<NUM>), the second stage <NUM> would pull up to the last eight (<NUM>) filtered audio features residing in the memory component <NUM> that were output from the first stage <NUM> during individual filtering of the audio features within a sequence of eight (<NUM>) input frames <NUM>. As the first stage <NUM> fills the corresponding memory component <NUM> to capacity, the memory locations <NUM> containing the oldest filtered audio features are overwritten (i.e., first in, first out). Thus, depending on the capacity of the memory component <NUM> at the neuron <NUM> or processing cell <NUM>, the second stage <NUM> is capable of remembering a number of past outputs processed by the first stage <NUM> of the corresponding SVDF processing cell <NUM>. Moreover, since the memory components <NUM> at the SVDF processing cells <NUM> are additive, the memory component <NUM> at each neuron <NUM> also includes the memory of each preceding neuron <NUM>, thus extending the overall receptive field of the memorized neural network <NUM>. As a result, the 3D SVDF layer(s) <NUM> and the sequentially-stacked SVDF layers <NUM> allow the neural network <NUM> to process only the audio features for one input time frame <NUM> (e.g., <NUM> milliseconds of audio data) at a time and incorporate a number of filtered audio features into the past that capture the fixed length of time necessary to capture the designated hotword in the multi-channel streaming audio <NUM>. By contrast, a neural network without memory would require its neurons <NUM> to process all of the audio feature frames covering the fixed length of time (e.g., <NUM> seconds of audio data) at once in order to determine the probability of the multi-channel streaming audio including the presence of the hotword, which drastically increases the overall size of the network. Moreover, while recurrent neural networks (RNNs) using long short-term memory (LSTM) provide memory, RNN-LSTMs cause the neurons to continuously update their state after each processing instance, in effect having an infinite memory, and thereby prevent the ability to remember a finite past number of processed outputs where each new output re-writes over a previous output (once the fixed-sized memory is at capacity). Put another way, SVDF networks do not recur the outputs into the state (memory), nor rewrite all the state with each iteration; instead, the memory keeps each inference run's state isolated from subsequent runs, instead pushing and popping in new entries based on the memory size configured for the layer.

Referring now to <FIG> and <FIG>, in some implementations, the memorized neural network <NUM> is trained on a plurality of training input audio sequences <NUM> (i.e., training samples) that each include a sequence of input frames <NUM> and labels <NUM> assigned to the input frames <NUM> for each channel <NUM> of the multi-channel streaming audio <NUM>. Each input frame <NUM> includes one or more respective audio features <NUM> characterizing phonetic components <NUM> of a hotword, and each label <NUM> indicates a probability that the one or more audio features <NUM> of a respective input frame <NUM> include a phonetic component <NUM> of the hotword. In the example shown, channel <NUM> of streaming audio <NUM> is provided, but it is understood that all other channels <NUM> of the multi-channel streaming audio <NUM> include similar training input audio sequences and components (e.g., respective audio feature <NUM> characterizing phonetic components <NUM> and labels <NUM> indicating a probability that the one or more audio features <NUM> includes a phonetic component <NUM> of the hotword). In some examples, the audio features <NUM> for each input frame <NUM> are converted from raw audio signals <NUM> of a channel <NUM> of the multi-channel audio stream <NUM> during a pre-processing stage <NUM> (i.e., a feature extraction or feature generation stage). The audio features <NUM> may include one or more log-filterbanks. Thus, the pre-processing stage may segment the audio stream channel <NUM> into the sequence of input frames <NUM> (e.g., <NUM> each), and generate separate log-filterbanks for each frame <NUM>. For example, each frame <NUM> may be represented by forty log-filterbanks. Each SVDF processing cell <NUM> of the input 3D SVDF layer <NUM> receives the sequence of input frames <NUM> from a respective channel <NUM>. Moreover, each successive layer (e.g., SVDF layers <NUM>) receives, as input, the concatenated filtered audio features <NUM> (i.e., output <NUM>) with respect to time. In some examples, the 3D SVDF input layer <NUM> includes multiple parallel SVDF processing cells <NUM> and each SVDF processing cell <NUM> is associated with a respective channel of each multi-channel training input audio sequence <NUM> and configured to receive the respective audio features <NUM> of each respective input frame <NUM> of the respective channel individually.

In the example shown (i.e., channel <NUM> of the multi-channel streaming audio <NUM>), each training input audio sequence <NUM> is associated with a training sample that includes an annotated utterance containing a designated hotword occurring within a fixed length of time (e.g., two seconds). The memorized neural network <NUM> may be trained on such a training input audio sequence <NUM> for each SVDF processing cell <NUM> of the input 3D SVDF layer <NUM>. For example, two SVDF processing cells <NUM> (i.e., for two channels <NUM> of multi-channel streaming audio <NUM>), the memorized neural network <NUM> may receive two training input audio sequences <NUM>. The memorized neural network <NUM> may also optionally be trained on annotated utterances <NUM> that do not include the designated hotword, or include the designated hotword but spanning a time longer than the fixed length of time, and thus, would not be falsely detected due to the fixed memory forgetting data outside the fixed length of time. In some examples, the fixed length of time corresponds to an amount of time that a typical speaker would take to speak the designated hotword to summon a user device <NUM> for processing spoken queries and/or voice commands. For instance, if the designated hotword includes the phrase "Hey Google" or "Ok Google", a fixed length of time set equal to two seconds is likely sufficient since even a slow speaker would generally not take more than two seconds to speak the designated phrase. Accordingly, since it is only important to detect the occurrence of the designated hotword within streaming audio <NUM> during the fixed length of time, the neural network <NUM> includes an amount of fixed memory that is proportional to the amount of audio to span the fixed time (e.g., two seconds). Thus, the fixed memory of the neural network <NUM> allows neurons <NUM> of the neural network to filter audio features <NUM> (e.g., log-filterbanks) from one input frame <NUM> (e.g., <NUM> time window) for each channel <NUM> of the streaming audio <NUM> at a time, while storing the most recent filtered audio features <NUM> spanning the fixed length of time and removing or deleting any filtered audio features <NUM> outside the fixed length of time from a current filtering iteration. Thus, if the neural network <NUM> has, for example, a memory depth of thirty-two (<NUM>), the first thirty-two (<NUM>) frames processed by the neural network <NUM> will fill the memory component <NUM> to capacity, and for each new output after the first <NUM>, the neural network <NUM> will remove the oldest processed audio feature from the corresponding memory location <NUM> of the memory component <NUM>.

Referring to <FIG>, for end-to-end training, training input audio sequence 500a includes labels <NUM> that may be applied to each input frame <NUM>. In some examples, when a training sample 500a contains the hotword, a target label <NUM> associated with a target score (e.g., '<NUM>') is applied to one or more input frames <NUM> that contain audio features <NUM> characterizing phonetic components <NUM> at or near the end of the hotword. For example, if the phonetic components <NUM> of the hotword "OK Google" are broken into: "ou", 'k', "el", "<silence>", 'g', 'u', 'g', '@', 'l', then target labels of the number '<NUM>' are applied to all input frames <NUM> that correspond to the letter 'l' (i.e. the last component <NUM> of the hotword), which are part of the required sequence of phonetic components <NUM> of the hotword. In this scenario, all other input frames <NUM> (not associated with the last phonetic component <NUM>) are assigned a different label (e.g., '<NUM>'). Thus, each input frame <NUM> includes a corresponding input feature-label pair <NUM>, <NUM>. The input features <NUM> are typically one-dimensional tensors corresponding to, for example, mel filterbanks or log-filterbanks, computed from the input audio over the input frame <NUM>. The labels <NUM> are generated from the annotated utterances 500a, where each input feature tensor <NUM> is assigned a phonetic class via a force-alignment step (i.e., a label of '<NUM>' is given to pairs corresponding to the last class belonging to the hotword, and '<NUM>' to all the rest). Thus, the training input audio sequence 500a includes binary labels assigned to the sequence of input frames. The annotated utterances 500a, or training input audio sequence 500a, correspond to the training samples <NUM> obtained from the annotated utterance pools <NUM> of <FIG>.

In another implementation, <FIG> includes a training input audio sequence 500b that includes labels <NUM> associated with scores that increase along the sequence of input frames <NUM> as the number of audio features <NUM> characterizing (matching) phonetic components <NUM> of the hotword progresses. For instance, when the hotword includes "Ok Google", the input frames <NUM> that include respective audio features <NUM> that characterize the first phonetic components, 'o' and 'k', have assigned labels <NUM> of 'l', while the input frames <NUM> that include respective audio features <NUM> characterizing the final phonetic component of 'l' have assigned labels <NUM> of '<NUM>'. The input frames <NUM> including respective audio features <NUM> characterizing the middle phonetic components <NUM> have assigned labels <NUM> of '<NUM>', '<NUM>', and '<NUM>'.

In additional implementations, the number of positive labels <NUM> increases. For example, a fixed amount of '<NUM>' labels <NUM> is generated, starting from the first frame <NUM> including audio features <NUM> characterizing to the final phonetic component <NUM> of the hotword. In this implementation, when the configured number of positive labels <NUM> (e.g., '<NUM>') is large, a positive label <NUM> may be applied to frames <NUM> that otherwise would have been applied a non-positive label <NUM> (e.g., '<NUM>'). In other examples, the start position of the positive label <NUM> is modified. For example, the label <NUM> may be shifted to start at either a start, mid-point, or end of a segment of frames <NUM> containing the final keyword phonetic component <NUM>. Still yet in other examples, a weight loss is associated with the input sequence. For example, weight loss data is added to the input sequence that allows the training procedure to reduce the loss (i.e. error gradient) caused by small mis-alignment. Specifically, with frame-based loss functions, a loss can be caused from either mis-classification or mis-alignment. To reduce the loss, the neural network <NUM> predicts both the correct label <NUM> and correct position (timing) of the label <NUM>. Even if the network <NUM> detected the keyword at some point, the result can be considered an error if it's not perfectly aligned with the given target label <NUM>. Thus, weighing the loss is particularly useful for frames <NUM> with high likelihood of mis-alignment during the force-alignment stage.

As a result of training using either of the training input audio sequences 500a, 500b of <FIG> and <FIG>, the neural network <NUM> is optimized (typically using cross-entropy (CE) loss) to output binary decision labels <NUM> indicating whether the hotword(s) are present in the streaming audio <NUM>. Referring now to <FIG>, each training sample <NUM> (i.e., each multi-channel training input audio sequence) includes a channel of training input audio sequence <NUM> for each SVDF processing cell <NUM> of the input 3D SVDF layer <NUM>. In some implementations, the training samples <NUM> include a first portion of training samples 502D (i.e., training input audio sequences) where each channel is a duplicate with each other channel of the multi-channel training input audio sequence. For example, a training sample 502D that includes two channels includes an identical training sequence for each channel. Each channel in a remaining second portion of training samples 502U may be unique to each other channel of the multi-channel training input audio sequence. For example, a training sample 502U that includes two channels would include a different training sequence for each channel (e.g., captured by two separate microphones).

Including a mix of training samples <NUM> that include both duplicated channels 502D and unique channels 502U ensures that the memorized neural network <NUM> is trained to accurately respond when the network <NUM> receives valid audio data from multiple channels <NUM> (i.e., multiple microphones <NUM>) and also when the network receives only a single valid channel <NUM> of audio. For example, a user device <NUM> may include only a single microphone <NUM> or one or more microphones <NUM> of a multi-microphone user device <NUM> may fail. In either case, it is desirable for the memorized neural network to still accurately detect the presence of the hotword in utterances <NUM>. By training the memorize neural network <NUM> on a portion of training samples 502D that provide only a single channel of unique audio (as each channel is a duplicate of each other channel), the memorized neural network <NUM> learns to accurately detect the presence of a hotword when provided with only a single channel of valid audio.

Referring now to <FIG>, schematic view <NUM> shows the neural network <NUM> that includes, for example, twelve layers, that are trained to produce acoustic posterior probabilities. In addition to the 3D SVDF layers <NUM>, the network <NUM> may, for example, include SVDF layers, dense layers, bottleneck layers, softmax layers, and/or other layers. The neural network <NUM> may be trained end-to-end. For example, the neural network <NUM> accepts features directly and uses the binary target label <NUM> (i.e., '<NUM>' or '<NUM>') outputs for use in training the network <NUM>. Such an end-to-end neural network <NUM> may use any topology. In some examples, the network <NUM> includes an encoder portion and a decoder portion.

Thus, the neural network <NUM> provides a small footprint while increasing accuracy and noise robustness using multiple channels of streaming audio captured by multiple independent microphones. The hotword detection system <NUM> requires only a single memorized neural network <NUM> for any number of audio channels <NUM>, thus significantly reducing required computing resources. The memorized neural network <NUM> is trained using a mix of duplicated and unique audio channels so that the network accurately detects the presence of a hotword when receiving both a single channel of audio and multiple channels of audio.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of detecting a hotword in multi-channel streaming audio <NUM>. The method <NUM>, at step <NUM>, includes receiving, at data processing hardware <NUM> of a user device <NUM>, a sequence of input frames <NUM> characterizing streaming multi-channel audio <NUM> captured by an array of microphones <NUM> in communication with the data processing hardware <NUM>. Each channel of the streaming multi-channel audio <NUM> includes respective audio features <NUM> captured by a separate dedicated microphone <NUM> in the array of microphones. For each input frame, the method <NUM>, at step <NUM>, includes processing, by the data processing hardware <NUM>, using a three-dimensional (3D) singular value decomposition filter (SVDF) input layer <NUM> of a memorized neural network <NUM>, the respective audio features <NUM> of each channel <NUM> of the streaming multi-channel audio 118in parallel. Also for each input frame, the method <NUM>, at step <NUM>, includes generating, by the data processing hardware <NUM>, using an intermediate layer <NUM> of the memorized neural network <NUM>, a corresponding multi-channel audio feature representation <NUM> based on a concatenation of the respective audio features <NUM> of each channel <NUM> of the streaming multi-channel audio <NUM>.

At step <NUM>, the method <NUM> also includes generating, by the data processing hardware <NUM>, using sequentially-stacked SVDF layers <NUM> of the memorized neural network <NUM>, a probability score <NUM> that indicates a presence of a hotword in the streaming multi-channel audio <NUM> based on the corresponding multi-channel audio feature representation <NUM> of each input frame <NUM>. The sequentially-stacked SVDF layers <NUM> include an initial SVDF layer <NUM> configured to receive the corresponding multi-channel audio feature representation <NUM> of each input frame <NUM> in sequence. At step <NUM>, the method <NUM> includes determining, by the data processing hardware <NUM>, whether the probability score <NUM> satisfies a hotword detection threshold, and at step <NUM>, when the probability score <NUM> satisfies the hotword detection threshold, initiating, by the data processing hardware <NUM>, a wake-up process on the user device <NUM> for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio <NUM>.

For example, it may be implemented as a standard server 900a or multiple times in a group of such servers 900a, as a laptop computer 900b, or as part of a rack server system 900c.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>) of a user device (<NUM>), a sequence of input frames (<NUM>) characterizing streaming multi-channel audio (<NUM>) captured by an array of microphones (<NUM>) in communication with the data processing hardware (<NUM>), each channel (<NUM>) of the streaming multi-channel audio (<NUM>) comprising respective audio features (<NUM>) captured by a separate dedicated microphone (<NUM>) in the array of microphones (<NUM>);
for each input frame (<NUM>):
processing, by the data processing hardware (<NUM>), using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer (<NUM>) of a memorized neural network (<NUM>), the respective audio features (<NUM>) of each channel (<NUM>) of the streaming multi-channel audio (<NUM>) in parallel; and
generating, by the data processing hardware (<NUM>), using an intermediate layer (<NUM>) of the memorized neural network (<NUM>), a corresponding multi-channel audio feature representation (<NUM>) based on a concatenation (<NUM>) of the respective audio features (<NUM>) of each channel (<NUM>) of the streaming multi-channel audio (<NUM>);
generating, by the data processing hardware (<NUM>), using sequentially-stacked SVDF layers (<NUM>) of the memorized neural network (<NUM>), a probability score (<NUM>) indicating a presence of a hotword in the streaming multi-channel audio (<NUM>) based on the corresponding multi-channel audio feature representation (<NUM>) of each input frame (<NUM>);
determining, by the data processing hardware (<NUM>), whether the probability score (<NUM>) satisfies a hotword detection threshold; and
when the probability score (<NUM>) satisfies the hotword detection threshold, initiating, by the data processing hardware (<NUM>), a wake-up process on the user device (<NUM>) for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio (<NUM>).