# Neural Network Exploration for Keyword Spotting on Edge Devices
## 2 Background
### Selected Neural Network Architectures
In short, a dense block is a series of alternating convolution and concatenation operations where the input to each convolution layer is the combined output of every preceding convolution layer [12]. Thus, each layer passes its own feature maps to all subsequent layers. Dense blocks do not reduce the dimensions of feature maps due to the presence of concatenation operations. Thus, transition layers are inserted between dense blocks, where each transition layer consists of a convolutional layer followed by pooling. The DenseNet architecture has two other variations: DenseNet-B and DenseNet-C [12]. DenseNet-B introduces bottleneck layers, which are inserted before convolution layers within dense blocks to reduce the number of feature maps and thus reduce the total number of weights. DenseNet-C introduces a compression factor \(\theta\) as part of transition blocks to reduce the number of feature maps passed between dense blocks. DenseNet-B and DenseNet-C may be combined to form DenseNet-BC to prevent the number of parameters from ballooning. Figure 2 shows examples of a dense block and a transition block in DenseNet. In these figures, \(W\), \(H\), and \(C\) in \(x\) (\(W,H,C\)) represent,respectively, the width, height, and the number of channels of the input matrix \(x\). The same notation applies to the output \(y\). Each convolution block, e.g., Convolution \((N,W,H,C)\), uses \(N\) convolution kernels, each with a size of width \((W)\times\) height \((H)\times\) channels \((C)\), and produces \(N\) output feature maps. \(B\) in Figure 2a represents the number of feature maps produced by the bottleneck layer in DenseNet-B. * Inception: The Inception architecture was designed to address two problems associated with increasing convolutional neural network size: the increase in the number of parameters and the amount of computational resources required to produce a result. These problems share a solution: sparsity, i.e., by replacing all fully connected layers with sparsely connected layers. However, due to limitations in modern computer architecture, sparse calculations are very inefficient. Thus, the Inception module was developed to approximate these sparse layers while maintaining reasonable efficiency on modern computers. An Inception module is composed of a series of parallel convolutional layers with different filter sizes (\(1\times 1,3\times 3,5\times 5\)) and a pooling layer [10]. With multiple filter sizes, the network is enhanced to capture features at different levels of abstraction. Chaining these modules together then forms an Inception neural network. The structure of an Inception module is depicted in Figure 3. Inception was originally designed for image classification and detection [10], and there is no known previous work applying it to keyword spotting. However, it is reasonable to apply Inception to the speech recognition problem, since a mel spectrogram is a pictorial representation of an audio recording. * CENet: To enable execution on embedded systems, CENet employs a residual bottleneck block. These bottleneck layers are connected to form a stage, and multiple stages can be chained together to form a complete network. As the bottleneck blocks do not decrease the size of the feature maps, a residual connection block is required to transition between stages. Each connection block decreases the number of internal feature maps by a factor of \(R\in(0,1]\) to lower resource requirements. These connection layers are inserted between stages of bottleneck blocks, forming a complete network. Figure 4 illustrates the structures of a residual bottleneck block and a residual con  
Figure 2: (**a**) Structure of a simple dense block with bottleneck layers and a growth rate of \(k\); (**b**) structure of a DenseNet transition block with compression factor \(\theta\). nection block in CENet. The CENet architecture was designed explicitly to perform keyword spotting on resource-constrained devices.