## 5 CNN-SSAE Neural Network
### Stacked Sparse Autoencoder  
Multiple SAEs are stacked together to form an SSAE. It is suitable for the fusion and compression of information in input data and can extract necessary features. The SAE is usually used for data compression and fusion. The name refers to the sparse restrictions that are added to each hidden layer. The previous layer contains more hidden cells than the latter layer, and the hidden features learned in the latter layer are more abstract. This structure is similar to the working state of the human brain.  
The SAE1 shown in Figure 8 is composed of an input layer, a hidden layer and an a output layer. The input data \(x1\) are mapped to a hidden laye1; this is called the encoding process. The hidden layer1 is remapped to the reconstructed data \(y1\); this is called the decoding process. This process should minimize the error function E from \(x1\) to \(y1\):  
\[E=\sum_{n=1}^{N}\|x_{n}-y_{n}\|^{2}+\beta\sum_{j=1}^{M}KL(\rho\|\hat{p}_{j}) \tag{8}\]  
where \(n\) denotes the number of samples, \(j\) denotes the number of hidden layers, \(\rho\) denotes the target value of the average activation degree, and \(\hat{p}_{j}\) denotes the average activation degree of the j-th layer.  
\[\hat{p}_{j}=\frac{1}{N}\sum_{n=1}^{N}f(W_{j}x+b_{j}) \tag{9}\]  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layers** & **Padding** & **Kernel Size** & **Strides** & **Output Shape** \\ \hline Input & 3-channel color picture & & [80,504,3] \\ Convolutional layer Con1 & SAME & [1,6,3,32] & [1,1] & [80,504,32] \\ Pooling layer Pool1 & SAME & [1,4] & [1,4] & [80,126,32] \\ Convolutional layer Con2 & SAME & [4,1,32,64] & [1,1] & [80,126,64] \\ Pooling layer Pool2 & SAME & [2,1] & [2,1] & [40,126,64] \\ Convolutional layer Con3 & SAME & [3,3,64,96] & [1,1] & [40,126,96] \\ Convolutional layer Con4 & VALID & [40,2,96,128] & [1,1] & [1,125,128] \\ Pooling layer Pool3 & Reshape the tensor into a single dimension & [125,128] \\ GAP & & Global Average Pooling & [128] \\ \hline \hline \end{tabular}
\end{table}
Table 1: CNN network architecture and parameter.  
Figure 7: The structure of the CNN.  
\(KL(\rho\|\hat{\rho}_{j})\) means \(KL\) distance (Kullback-Leibler divergence):  
\[KL(\rho\|\hat{\rho}_{j})=\rho\log(\frac{\rho}{\rho_{j}})+(1-\rho)\log(\frac{1- \rho}{1-\hat{\rho}_{j}}) \tag{10}\]  
When training the network, the parameters should be constantly adjusted to minimum \(\beta\). A value of \(\rho\) closer to 0 corresponds to a smaller average activation degree of the middle layer \(\hat{\rho}_{j}\). In addition, it is necessary to constantly adjust the connection weight \(W_{j}\) and bias \(b_{j}\) to minimum the error function \(E\).  
In the SSAE pre-training show in Figure 8, the weights and biases of each layer are \(W_{j}\) and \(b_{j}\), the adjusted parameters are \(\delta_{j}\) and \(\delta^{\prime}_{j^{\prime}}\) and the weights and biases after fine-tuning are \(W^{*}_{j}=W_{j}+\delta_{j}\) and \(b^{*}_{j}=b_{j}+\delta^{\prime}_{j^{\prime}}\). SSAE is formed by stacking multiple layers. First, the 128-dimensional deep features of the spectrogram are as input layer of \(SAE1\), whose hidden layer1 output is the 118-dimensional vector \(h^{(1)}\), where \(h^{(1)}=sigmoid(b^{*}_{1}+\sum W^{*}_{1}x)\). Second, hidden layer1 is regarded as the input of \(SAE2\), whose hidden layer2 output is the 108-dimensional vector \(h^{(2)}\), \(h^{(2)}=sigmoid(b^{*}_{2}+\sum W^{*}_{2}h^{(1)})\). Third, hidden layer2 is regarded as the input of \(SAE3\), whose hidden layer output is the 98-dimensional vector \(h^{(3)}\), where \(h^{(3)}=sigmoid(b^{*}_{3}+\sum W^{*}_{3}h^{(2)})\). Finally, the deep optimized features \(h^{(3)}\) are fed into the softmax layer for recognition.