This table also shows how 1stage outperforms 2stage for small size models, and presents similar FR rates as Baseline_1850K on clean conditions.

End-to-end model setup	The end-to-end system (prefix E2E) uses the DNN topology depicted in Figure FIGREF8 . We present results with 3 distinct size configurations (infixes 700K, 318K, and 40K) each representing the number of approximate parameters, and 2 types of training recipes (suffixes 1stage and 2stage) corresponding to end-to-end and encoder+decoder respectively, as described in UID7 . The input to all DNNs consist of a sequence with INLINEFORM0 frames of left and INLINEFORM1 frames of right context; each with a stride of INLINEFORM2 . More specifically, the E2E_700K model uses INLINEFORM3 nodes in the first 4 SVDF layers, each with a memory INLINEFORM4 , with intermediate bottleneck layers each of size 64; the following 3 SVDF layers have INLINEFORM5 nodes, each with a memory INLINEFORM6 . This model performs 350K multiply-accumulate operations per inference (every 20ms of streaming audio).

During the second stage of training the encoder parameters are frozen, such that only the decoder is trained. This recipe useful on models that tend to overfit to parts of the training set. End-to-end. In this option, we train the DNN end-to-end directly, with the sequences from SECREF5 . The DNN may use any topology, but we use that of the encoder+decoder, except for the intermediate encoder softmax. See Figure FIGREF8 . Similar to the encoder+decoder recipe, we can also initialize the encoder part with a pre-trained model, and use an adaptation rate INLINEFORM0 to tune how much the encoder part is being adjusted (e.g. a rate of 0 is equivalent to the encoder+decoder recipe). This end-to-end pipeline, where the entirety of the topology's parameters are adjusted, tends to outperform the encoder+decoder one, particularly in smaller sized models which do not tend to overfit.

Another benefit is that due to the explicit sizing of the receptive field it allows for a fine grained control over how much to remember from the past. This has resulted in SVDF outperforming RNN-LSTMs, which do not benefit from, and are potentially hurt by, paying attention to theoretically infinite past. It also avoids having complex logic to reset the state every few seconds as in  .

Conclusion	We presented a system for keyword spotting that by combining an efficient topology and two types of end-to-end training can significantly ourperform previous appraoches, at a much lower cost of size and computation. We specifically show how it beats the performance of a setup taken from  with models over 5 times smaller, and even get close to the same performance with a model over 40 times smaller. Our approach provides further benefits of not requiring anything other than a front-end and a neural network to perform the detection, and thus it is easier to extend to newer keywords and/or fine-tune with new training data. Future work includes exploring other loss-functions, as well as generalizing multi-channel support.

Thus for each node INLINEFORM1 , the input INLINEFORM2 goes through the feature filter INLINEFORM3 , and the resulting scalar output gets concatenated to those INLINEFORM4 computed in previous inference steps. The memory is either initialized to zeros during training for the first INLINEFORM5 inferences. Finally the time filter INLINEFORM6 is applied to them. This is how stateful networks work, where the layer is able to memorize the past within its state. Different from typical recurrent approaches though, and other types of stateful layers  , the SVDF does not recur the outputs into the state (memory), nor rewrites the entirety of the state with each iteration. Instead, the memory keeps each inference's state isolated from subsequent runs, just pushing new entries and popping old ones based on the memory size INLINEFORM7 configured for the layer. This also means that by stacking SVDF layers we are extending the receptive field of the network. For example, a DNN with INLINEFORM8 stacked layers, each with a memory of INLINEFORM9 , means that the DNN is taking into account inputs as old as INLINEFORM10 .

End-to-End Streaming Keyword Spotting	We present a system for keyword spotting that, except for a frontend component for feature generation, it is entirely contained in a deep neural network (DNN) model trained"end-to-end"to predict the presence of the keyword in a stream of audio. The main contributions of this work are, first, an efficient memoized neural network topology that aims at making better use of the parameters and associated computations in the DNN by holding a memory of previous activations distributed over the depth of the DNN. The second contribution is a method to train the DNN, end-to-end, to produce the keyword spotting score. This system significantly outperforms previous approaches both in terms of quality of detection as well as size and computation.

Of those components, it is the last two that make use of DNNs along with a wide variety of decoding implementations. They range from traditional approaches that make use of a Hidden Markov Model (HMM) to characterize acoustic features from a DNN into both "keyword" and "background" (i.e. non-keyword speech and noise) classes  ,  ,  ,  ,  . Simpler derivatives of that approach perform a temporal integration computation that verifies the outputs of the acoustic model are high in the right sequence for the target keyword in order to produce a single detection likelyhood score  ,  ,  ,  ,  . Other recent systems make use of CTC-trained DNNs –typically recurrent neural networks (RNNs)  , or even sequence-to-sequence trained models that rely on beam search decoding  . This last family of systems is the closest to be considered end-to-end, however they are generally too computationally complex for many embedded applications. Optimizing independent components, however, creates added complexities and is suboptimal in quality compared to doing it jointly. Deployment also suffers due to the extra complexity, making it harder to optimize resources (e.g.

Method to train the end-to-end neural network	The goal of our end-to-end training is to optimize the network to produce the likelihood score, and to do so as precisely as possible. This means have a high score right at the place where the last bit of the keyword is present in the streaming audio, and not before and particularly not much after (i.e. a "spiky" behaviour is desirable). This is important since the system is bound to an operating point defined by a threshold (between 0 and 1) that is choosen to strike a balance between false-accepts and false-rejects, and a smooth likelyhood curve would add variability to the firing point. Moreover, any time between the true end of the keyword and the point where the score meets the threshold will become latency in the system (e.g. the "assistant" will be slow to respond). A common drawback of CTC-trained RNNs  we aim to avoid. We generate input sequences composed of pairs < INLINEFORM0 , INLINEFORM1 >.

These 2 models, E2E_40K_1stage and E2E_40K_2stage, also explore the capacity of end-to-end training (1stage) versus encoder+decoder training (2stage). As can be appreciated in the ROC curves, 1stage training outperforms 2stage training on all conditions, but particularly on both "clean" environments where it gets fairly close to the performance of the baseline setup. That is a significant achievement considering E2E_40K_1stage has 2.3% the parameters and performs 3.2% the computations of Baseline_1850K. Table TABREF13 compares the recognition quality of all setups by fixing on a very low false-accept rate of 0.1 FA per hour on a dataset containing only negative (i.e. non-keyword) utterances. Thus the table shows the false-reject rates at that operating point. Here we can appreciate similar trends as those described above: the 2 largest end-to-end models outperforms the baseline across all datasets, reducing FR rate about 40% on the clean conditions and 40%-20% on the other 2 sets depending on the model size.

Introduction	Keyword detection is like searching for a needle in a haystack: the detector must listen to continuously streaming audio, ignoring nearly all of it, yet still triggering correctly and instantly. In the last few years, with the advent of voice assistants, keyword spotting has become a common way to initiate a conversation with them (e.g. "Ok Google", "Alexa", or "Hey Siri"). As the assistant use cases spread through a variety of devices, from mobile phones to home appliances and further into the internet-of-things (IoT) –many of them battery powered or with restricted computational capacity, it is important for the keyword spotting system to be both high-quality as well as computationally efficient. Neural networks are core to the state of-the-art keyword spotting systems. These solutions, however, are not developed as a single deep neural network (DNN). Instead, they are traditionally comprised of different subsystems, independently trained, and/or manually designed. For example, a typical system is composed by three main components: 1) a signal processing frontend, 2) an acoustic encoder, and 3) a separate decoder.

This approach works very well for streaming execution, like in speech, text, and other sequential processing, where we constantly process new inputs from a large, possibly infinite sequence but do not want to attend to all of it. An implementation is available at  . This layer topology offers a number of benefits over other approaches. Compared with the convolutions use in  , it allows finer-grained control of the number of parameters and computation, given that the SVDF is composed by several relatively small filters. This is useful when selecting a tradeoff between quality, size and computation. Moreover, because of this characteristic, the SVDF allows creating very small networks that outperform other topologies which operate at larger granularity (e.g. our first stage, always-on network has about 13K parameters  ). The SVDF also pairs very well with linear “bottleneck” layers to significantly reduce the parameter count as in  ,  , and more recently in  . And because it allows for creating evenly sized deep networks, we can insert them throughout the network as in Figure FIGREF8 .

Baseline model setup	Our baseline system (Baseline_1850K) is taken from  . It consists of a DNN trained to predict subword targets within the keywords. The input to the DNN consists of a sequence with INLINEFORM0 frames of left and INLINEFORM1 frames of right context; each with a stride of INLINEFORM2 . The topology consists of a 1-D convolutional layer with 92 filters (of shape 8x8 and stride 8x8), followed by 3 fully-connected layers with 512 nodes and a rectified linear unit activation each. A final softmax output predicts the 7 subword targets, obtained from the same forced alignment process described in SECREF5 . This results in the baseline DNN containing 1.7M parameters, and performing 1.8M multiply-accumulate operations per inference (every 30ms of streaming audio). A keyword spotting score between 0 and 1 is computed by first smoothing the posterior values, averaging them over a sliding window of the previous 100 frames with respect to the current INLINEFORM3 ; the score is then defined as the largest product of the smoothed posteriors in the sliding window as originally proposed in  .

The E2E_318K model uses INLINEFORM7 nodes in the first 4 SVDF layers, each with a memory INLINEFORM8 , with intermediate bottleneck layers each of size 64; the remainder layers are the same as E2E_700K. This model performs 159K multiply-accumulate operations per inference. Finally, the E2E_40K model uses INLINEFORM9 nodes in the first 4 SVDF layers, each with a memory INLINEFORM10 , with intermediate bottleneck layers each of size 32; the remainder layers are the same as the other two models. This model performs 20K multiply-accumulate operations per inference.

This is with the intetion of balancing the amount of negative and positive examples, in the same spirit as  . This proved important to make training stable, as otherwise the amount of negative examples overpowered the positive ones. The end-to-end training uses a simple frame-level cross-entropy (CE) loss that for the feature vector INLINEFORM0 is defined by INLINEFORM1 , where INLINEFORM2 are the parameters of the network, INLINEFORM3 the INLINEFORM4 th output of the final softmax. Our training recipe uses asynchronous stochastic gradient descent (ASGD) to produce a single neural network that can be fed streaming input features and produce a detection score. We propose two options to this recipe: Encoder+decoder. A two stage training procedure where we first train an acoustic encoder, as in  ,  ,  , and then a decoder from the outputs of the encoder (rather than filterbank energies) and the labels from SECREF5 . We do this in a single DNN by creating a final topology that is composed of the encoder and its pre-trained parameters (including the softmax), followed by the decoder. See Figure FIGREF8 .

Results	Our goal is to compare the efectiviness of the proposed approach against the baseline system described in  . We evaluate the false-reject (FR) and false-accept (FA) tradeoff across several end-to-end models of distinct sizes and computational complexities. As can be seen in the Receiver Operating Characteristic (ROC) curves in Figure FIGREF14 , the 2 largest end-to-end models, with 2-stage training, significantly outperform the recognition quality of the much larger and complex Baseline_1850K system. More specifically, E2E_318K_2stage and E2E_700K_2stage show up to 60% relative FR rate reduction over Baseline_1850K in most test conditions. Moreover, E2E_318K_2stage uses only about 26% of the computations that Baseline_1850K uses (once normalizing their execution rates over time), but still shows significant improvements. We also explore end-to-end models at a size that, as described in  , is small enough, in both size and computation, to be executed continuously with very little power consumption.

processing power and memory consumption). The system described in this paper addresses those concerns by learning both the encoder and decoder components into a single deep neural network, jointly optimizing to directly produce the detection likelyhood score. This system could be trained to subsume the signal processing frontend as well as in  ,  , but it is computationally costlier to replace highly optimized fast fourier transform implementations with a neural network of equivalent quality. However, it is something we consider exploring in the future. Overall, we find this system provides state of the art quality across a number of audio and speech conditions compared to a traditional, non end-to-end baseline system described in  . Moreover, the proposed system significantly reduces the resource requirements for deployment by cutting computation and size over five times compared to the baseline system. The rest of the paper is organized as follows. In Section SECREF2 we present the architecture of the keyword spotting system; in particular the two main contributions of this work: the neural network topology, and the end-to-end training methodology. Next, in Section SECREF3 we describe the experimental setup, and the results of our evaluations in Section SECREF4 , where we compare against the baseline approach of  .

Where INLINEFORM2 is a 1D tensor corresponding to log-mel filter-bank energies produced by a front-end as in  ,  ,  , and INLINEFORM3 is the class label (one of INLINEFORM4 ). Each tensor INLINEFORM5 is first force-aligned from annotated audio utterances, using a large LVCSR system, to break up the components of the keyword  . For example, "ok google" is broken into: "ou", "k", "eI", "<silence>", "g", "u", "g", "@", "l". Then we assign labels of 1 to all sequence entries, part of a true keyword utterance, that correspond to the last component of the keyword ("l" in our "ok google" example). All other entries are assigned a label of 0, including those that are part of the keyword but that are not its last component. See Figure FIGREF6 . Additionally, we tweak the label generation by adding a fixed amount of entries with a label of 1, starting from the first vector INLINEFORM6 corresponding to the final keyword component.

Experimental setup	In order to determine the effectiveness of our approach, we compare against a known keyword spotting system proposed in  . This section describes the setups used in the results section.

Front-end	Both setups use the same front-end, which generates 40-dimensional log-mel filter-bank energies out of 30ms windows of streaming audio, with overlaps of 10ms. The front-end can be queried to produce a sequence of contiguous frames centered around the current frame INLINEFORM0 . Older frames are said to form the left context INLINEFORM1 , and newer frames form the right context INLINEFORM2 . Additionally, the sequences can be requested with a given stride INLINEFORM3 .

Efficient memoized neural network topology	We make use of a type of neural network layer topology called SVDF (single value decomposition filter), originally introduced in  to approximate a fully connected layer with a low rank approximation. As proposed in  and depicted in equation EQREF2 , the activation INLINEFORM0 for each node INLINEFORM1 in the rank-1 SVDF layer at a given inference step INLINEFORM2 can be interpreted as performing a mix of selectivity in time ( INLINEFORM3 ) with selectivity in the feature space ( INLINEFORM4 ) over a sequence of input vectors INLINEFORM5 of size INLINEFORM6 . DISPLAYFORM0  This is equivalent to performing, on an SVDF layer of INLINEFORM0 nodes, INLINEFORM1 1-D convolutions of the feature filters INLINEFORM2 (by "sliding" each of the INLINEFORM3 filters on the input feature frames, with a stride of INLINEFORM4 ), and then filtering each of INLINEFORM5 output vectors (of size INLINEFORM6 ) with the time filters INLINEFORM7 . A more general and efficient interpretation, depicted in Figure FIGREF3 , is that the layer is just processing a single input vector INLINEFORM0 at a time.

End-to-End system	This paper proposes a new end-to-end keyword spotting system that by subsuming both the encoding and decoding components into a single neural network can be trained to produce directly an estimation (i.e. score) of the presence of a keyword in streaming audio. The following two sections cover the efficient memoized neural network topology being utilized, as well as the method to train the end-to-end neural network to directly produce the keyword spotting score.

Dataset	The training data for all experiments consists of 1 million anonymized hand-transcribed utterances of the keywords "Ok Google" and "Hey Google", with an even distribution. To improve robustness, we create "multi-style" training data by synthetically distorting the utterances, simulating the effect of background noise and reverberation. 8 distorted utterances are created for each original utterance; noise samples used in this process are extracted from environmental recordings of everyday events, music, and Youtube videos. Results are reported on four sets representative of various environmental conditions: Clean non-accented contains 170K non-accented english utterances of the keywords in "clean" conditions, plus 64K samples without the keywords (1K hours); Clean accented has 153K english utterances of the keywords with Australian, British, and Indian accents (also in "clean" conditions), plus 64K samples without the keywords (1K hours); High pitched has 1K high pitched utterances of the keywords, and 64K samples without them (1K hours); Query logs contains 110K keyword and 21K non-keyword utterances, collected from anonymized voice search queries.

This last set contains background noises from real living conditions.

Finally, we conclude with a discussion of our findings in Section SECREF5 .

Question: How many parameters does the presented model have?
Answer: (infixes 700K, 318K, and 40K) each representing the number of approximate parameters