Introduction
Speech processing enables natural communication with smart phones or smart home assistants, e.g., Amazon Echo, Google Home. However, continuously performing speech recognition is not energy-efficient and would drain batteries of smart devices. Instead, most speech recognition systems passively listen for utterances of certain wake words such as “Ok Google", “Hey Siri", “Alexa", etc. to trigger the continuous speech recognition system on demand. This task is referred to as keyword spotting (KWS). There are also uses of KWS where a view simple speech commands (e.g. “on", “off") are enough to interact with a device such as a voice-controlled light bulb.
Conventional hybrid approaches to KWS first divide their audio signal in time frames to extract features, e.g., Mel Frequency Cepstral Coefficients (MFCC). A neural net then estimates phoneme or state posteriors of the keyword Hidden Markov Model in order to calculate the keyword probability using a Viterbi search. In recent years, end-to-end architectures gained traction that directly classify keyword posterior probabilites based on the previously extracted features, e.g., BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4. Typical application scenarios imply that the device is powered by a battery, and possesses restricted hardware resources to reduce costs. Therefore previous works optimized towards memory footprint and operations per second. In contrast to this, we tune our neural network towards energy conservation in microcontrollers motivated by obervations on power consumption, as detailed in Sec. SECREF4. To extract meaningful and representative features from raw audio, our architecture uses parametrized Sinc-convolutions (SincConv) from SincNet proposed by Ravianelli et al. BIBREF5. We use Depthwise Separable Convolutions (DSConv) BIBREF6, BIBREF7 that preserve time-context information while at the same time compare features in different channels. To further reduce the number of network parameters, which is key for energy efficiency, we group DSConv-layers, a technique we refer to as Grouped DSConv (GDSConv). Our key contributions are:
We propose a neural network architecture tuned towards energy efficiency in microcontrollers grounded on the observation that memory access is costly, while computation is cheap BIBREF8.
Our keyword-spotting network classifies on raw audio employing SincConvs while at the same time reducing the number of parameters using (G)DSConvs.
Our base model with 122k parameters performs with the state-of-the-art accuracy of $96.6\%$ on the test set of Google’s Speech Commands dataset, on par with TC-ResNet BIBREF3 that has 305k parameters and requires separate preprocessing. Our low-parameter model achieves $96.4\%$ with only 62k parameters.
Related Work
Recently, CNNs have been successfully applied to KWS BIBREF1, BIBREF2, BIBREF3. Zhang et al. evaluated different neural network architectures (such as CNNs, LSTMs, GRUs) in terms of accuracy, computational operations and memory footprint as well as their deployment on embedded hardware BIBREF1. They achieved their best results using a CNN with DSConvs. Tang et al. explored the use of Deep Residual Networks with dilated convolutions to achieve a high accuracy of $95.8\%$ BIBREF2, while keeping the number of parameters comparable to BIBREF1. Choi et al. build on this work as they also use a ResNet-inspired architecture. Instead of using 2D convolution over a time-frequency representation of the data they convolve along the time dimension and treat the frequency dimension as channels BIBREF3.
This bears similarities with our approach as we are using 1D convolution along the time dimension as well. However, all the approaches mentioned classify from MFCCs or similar preprocessed features. Our architecture works directly on raw audio signals. There is a recent trend towards using CNNs on raw audio data directly BIBREF5, BIBREF9, BIBREF10, BIBREF11. Ravanelli et al. present an effective method of processing raw audio with CNNs, called SincNet. Kernels of the first convolutional layer are restricted to only learn shapes of parametrized sinc functions. This method was first introduced for Speaker Recognition BIBREF5 and later also used for Phoneme Recognition BIBREF9.
To the best of our knowledge, we are the first to apply this method to the task of KWS. The first convolutional layer of our model is inspired by SincNet and we combine it with DSCconv. DSCconvs have first been introduced in the domain of Image Processing BIBREF7, BIBREF12 and have been applied to other domains since: Zhang et al. applied DSCconv to KWS BIBREF1. Kaiser et al. used DSConv for neural machine translation BIBREF6. They also introduce the “super-separable” convolution, a DSConv that also uses grouping and thus reduces the already small number of parameters of DSConv even further. A similar method is used by ShuffleNet where they combine DSConv with grouping and channel shuffling BIBREF13. The idea of Grouped Convolutions was first used in AlexNet BIBREF14 to reduce parameters and operations and to enable distributed computing of the model over multiple GPUs. We denominate the combination of grouping and DSconv as GDSConv in our work and use it for our smallest model.
Model ::: Keyword-Spotting on Battery-Powered Devices
Typical application scenarios for smart devices imply that the device is powered by a battery, and possesses restricted hardware resources. The requirements for a KWS system in these scenarios are (1) very low power consumption to maximize battery life, (2) real-time or near real-time capability, (3) low memory footprint and (4) high accuracy to avoid random activations and to ensure responsiveness.
Regarding real-time capability, our model is designed to operate on a single-core microcontroller capable of 50 MOps per second BIBREF1. We assume that in microcontrollers the memory consumption of a KWS neural network is associated with its power consumption: Reading memory values contributes most to power consumption which makes re-use of weights favorable. While in general large memory modules leak more power than small memory modules, one read operation from RAM costs far more energy than the corresponding multiply-and-accumulate computation BIBREF15, BIBREF8. In addition to the parameter-reducing approach in this work, further steps may be employed to reduce power consumption such as quantization, model compression or optimization strategies regarding dataflows that depend on the utilized hardware platform BIBREF15, BIBREF8, BIBREF16, BIBREF17.
Model ::: Feature Extraction using SincConvs
SincNet BIBREF5 classifies on raw audio by restricting the filters of the first convolutional layer of a CNN to only learn parametrized sinc functions, i.e., $\operatorname{sinc}⁡(x)=\sin (x)/x$. One sinc function in the time domain represents a rectangular function in the spectral domain, therefore two sinc functions can be combined to an ideal band-pass filter:
Performing convolution with such a filter extracts the parts of the input signal that lie within a certain frequency range. SincNet combines Sinc-convolutions with CNNs; as we only use the feature extraction layer of this architecture, we label this layer as SincConv to establish a distinction to SincNet.
Compared to one filter of a regular CNN, the number of parameters is derived from its kernel width, e.g., $k=400$ BIBREF10. Sinc-convolutions only require two parameters to derive each filter, the lower and upper cut-off frequencies ($f_1,f_2$), resulting in a small memory footprint. SincConv filters are initialized with the cutoff frequencies of the mel-scale filter bank and then further adjusted during training. Fig. FIGREF7 visualizes this adjustment from initialization to after training. SincConv filter banks can be easily interpreted, as the two learned parameter correspond to a specific frequency band. Fig. FIGREF8 visualizes how a SincConv layer with 7 filters processes an audio sample containing the word “yes”.
Model ::: Low-Parameter GDSConv Layers
DSConv have been successfully applied to the domain of computer vision BIBREF7, BIBREF12, neural translation BIBREF6 and KWS BIBREF1. Fig. FIGREF10 provides an overview of the steps from a regular convolution to the GDSConv.
The number of parameters of one DSConv layer amounts to $N_{\text{DSConv}}=k\cdot c_{in}+c_{in}\cdot c_{out}$ with the kernel size $k$ and the number of input and output channels $c_{in}$ and $c_{out}$ respectively; the first summand is determined by the depthwise convolution, the second summand by the pointwise convolution BIBREF6. In our model configuration, the depthwise convolution only accounts for roughly $5\%$ of parameters in this layer, the pointwise for $95\%$. We therefore reduced the parameters of the pointwise convolution using grouping by a factor $g$ to $N_{\text{GDSConv}}=k\cdot c_{in}+\frac{c_{in}\cdot c_{out}}{g}$, rather than the parameters in the depthwise convolution. To allow information exchange between groups we alternate the number of groups per layer, namely 2 and 3, as proposed in BIBREF6.
Model ::: Two Low-Parameter Architectures
The SincConv as the first layer extracts features from the raw input samples, as shown in Fig. FIGREF12. As non-linearity after the SincConv we opt to use log-compression, i.e., $y=\log (\operatorname{abs}(x)+1)$, instead of a common activation function (e.g., ReLU). This has also shown to be effective in other CNN architectures for raw audio processing BIBREF10, BIBREF11. Five (G)DSConv layers are then used to process the features further: The first layer has a larger kernel size and scales the number of channels to 160. The other four layers have each 160 input and output channels. Each (G)DSConv block contains the (G)DSConv layer, batch normalization BIBREF19 and spatial dropout BIBREF20 for regularization, as well as average pooling to reduce temporal resolution. After the (G)DSConv blocks, we use global average pooling to receive a 160-element vector that can be transformed to class posteriors using a Softmax layer to classify 12 classes, i.e., 10 keywords as well as a class for unknown and for silence.
The low-parameter model is obtained by grouping the DSConv layers with an alternating number of groups between 2 and 3. For the configuration shown in Fig. FIGREF12, the base model has 122k parameters. After grouping, the number of parameters is reduced to a total of 62k.
Evaluation ::: Training on the Speech Commands Dataset
We train and evaluate our model using Google's Speech Commands data set BIBREF18, an established dataset for benchmarking KWS systems. The first version of the data set consists of 65k one-second long utterances of 30 different keywords spoken by 1881 different speakers. The most common setup consists of a classification of 12 classes: “yes", “no", “up", “down", “left", “right", “on", “off", “stop", “go", unknown, or silence. The remaining 20 keywords are labeled as unknown, samples of provided background noise files as silence. To ensure the benchmark reproducibility, a separate test set was released with a predefined list of samples for the unknown and the silence class. The second version of the dataset contains 105k samples and five additional keywords BIBREF18. However, previous publications on KWS reported only results on the first version, therefore we focused on the first version and additionally report testing results on version 2 of the dataset.
Every sample from the training set is used in training, this leads to a class imbalance as there are much more samples for unknown. Class weights in the training phase assign a lower weight to samples labeled as unknown such that the impact on the model is proportional to the other classes. This way, the model can see more unknown word samples during training without getting biased. Our model is trained for 60 epochs with the Adam optimizer BIBREF21 with an initial learning rate of 0.001 and learning rate decay of 0.5 after 10 epochs; the model with the highest validation accuracy is saved to evaluate accuracy on the test set.
Evaluation ::: Results and Discussion
The base model composed of DSConv layers without grouping achieves the state-of-the-art accuracy of 96.6% on the Speech Commands test set. The low-parameter model with GDSConv achieves almost the same accuracy of 96.4% with only about half the parameters. This validates the effectiveness of GDSConv for model size reduction. Table TABREF15 lists these results in comparison with related work. Compared to the DSConv network in BIBREF1, our network is more efficient in terms of accuracy for a given parameter count. Their biggest model has a 1.2% lower accuracy than our base model while having about 4 times the parameters. Choi et al. BIBREF3 has the most competitive results while we are still able to improve upon their accuracy for a given number of parameters. They are using 1D convolution along the time dimension as well which may be evidence that this yields better performance for audio processing or at least KWS.
As opposed to previous works, our architecture does not use preprocessing to extract features, but is able to extract features from raw audio samples with the SincConv layer. That makes it possible to execute a full inference as floating point operations, without requiring additional hardware modules to process or transfer preprocessed features. Furthermore, we deliberately opted to not use residual connections in our network architecture, considering the memory overhead and added difficulty for hardware acceleration modules.
For future comparability, we also trained and evaluated our model on the newer version 2 of the Speech Commands data set; see Table TABREF16 for results. On a side note, we observed that models trained on version 2 of the Speech Commands dataset tend to perform better on both the test set for version 2 and the test set for version 1 BIBREF18.
Conclusion
Always-on, battery-powered devices running keyword spotting require energy efficient neural networks with high accuracy. For this, we identified the parameter count in a neural network as a main contributor to power consumption, as memory accesses contribute far more to power consumption than the computation. Based on this observation, we proposed an energy efficient KWS neural network architecture by combining feature extraction using SincConvs with GDSConv layers.
Starting with the base model composed of DSConvs that have already less parameters than a regular convolution, we achieved state-of-the-art accuracy on Google's Speech Commands dataset. We further reduce the number of parameters by grouping the convolutional channels to GDSConv, resulting in a low-parameter model with only 62k parameters.

Question: Do they compare executionttime of their model against other models?
Answer: No

Question: What is the memory footprint decrease of their model in comparison to other models?
Answer: Unanswerable