Abstract: We present a system for keyword spotting that, except for a frontend component for feature generation, it is entirely contained in a deep neural network (DNN) model trained"end-to-end"to predict the presence of the keyword in a stream of audio. The main contributions of this work are, first, an efficient memoized neural network topology that aims at making better use of the parameters and associated computations in the DNN by holding a memory of previous activations distributed over the depth of the DNN. The second contribution is a method to train the DNN, end-to-end, to produce the keyword spotting score. This system significantly outperforms previous approaches both in terms of quality of detection as well as size and computation.

Introduction: Keyword detection is like searching for a needle in a haystack: the detector must listen to continuously streaming audio, ignoring nearly all of it, yet still triggering correctly and instantly. In the last few years, with the advent of voice assistants, keyword spotting has become a common way to initiate a conversation with them (e.g. "Ok Google", "Alexa", or "Hey Siri"). As the assistant use cases spread through a variety of devices, from mobile phones to home appliances and further into the internet-of-things (IoT) –many of them battery powered or with restricted computational capacity, it is important for the keyword spotting system to be both high-quality as well as computationally efficient. Neural networks are core to the state of-the-art keyword spotting systems. These solutions, however, are not developed as a single deep neural network (DNN). Instead, they are traditionally comprised of different subsystems, independently trained, and/or manually designed. For example, a typical system is composed by three main components: 1) a signal processing frontend, 2) an acoustic encoder, and 3) a separate decoder. Of those components, it is the last two that make use of DNNs along with a wide variety of decoding implementations. They range from traditional approaches that make use of a Hidden Markov Model (HMM) to characterize acoustic features from a DNN into both "keyword" and "background" (i.e. non-keyword speech and noise) classes BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . Simpler derivatives of that approach perform a temporal integration computation that verifies the outputs of the acoustic model are high in the right sequence for the target keyword in order to produce a single detection likelyhood score BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 . Other recent systems make use of CTC-trained DNNs –typically recurrent neural networks (RNNs) BIBREF10 , or even sequence-to-sequence trained models that rely on beam search decoding BIBREF11 . This last family of systems is the closest to be considered end-to-end, however they are generally too computationally complex for many embedded applications. Optimizing independent components, however, creates added complexities and is suboptimal in quality compared to doing it jointly. Deployment also suffers due to the extra complexity, making it harder to optimize resources (e.g. processing power and memory consumption). The system described in this paper addresses those concerns by learning both the encoder and decoder components into a single deep neural network, jointly optimizing to directly produce the detection likelyhood score. This system could be trained to subsume the signal processing frontend as well as in BIBREF2 , BIBREF12 , but it is computationally costlier to replace highly optimized fast fourier transform implementations with a neural network of equivalent quality. However, it is something we consider exploring in the future. Overall, we find this system provides state of the art quality across a number of audio and speech conditions compared to a traditional, non end-to-end baseline system described in BIBREF13 . Moreover, the proposed system significantly reduces the resource requirements for deployment by cutting computation and size over five times compared to the baseline system. The rest of the paper is organized as follows. In Section SECREF2 we present the architecture of the keyword spotting system; in particular the two main contributions of this work: the neural network topology, and the end-to-end training methodology. Next, in Section SECREF3 we describe the experimental setup, and the results of our evaluations in Section SECREF4 , where we compare against the baseline approach of BIBREF13 . Finally, we conclude with a discussion of our findings in Section SECREF5 .

Question: How many rules had to be defined?
Answer: WikiSQL - 2 rules (SELECT, WHERE)
SimpleQuestions - 1 rule
SequentialQA - 3 rules (SELECT, WHERE, COPY)