## 2 Related Works
### DNN Algorithm
#### 2.2.3 Sparse Auto-Encoder (SAE)  
The default auto-encoder attempts to learn feature approximation, and the function of the auto-encoder of \(x^{\prime}\) is close to the raw input data of \(x\). It is seems that the identity function is able to learn most features of the input data. However, the number of units in each layer of the neural network is constrained, with \(x^{\prime}\) as the minimum original size. When the number of hidden units increases, runtime efficiency decreases for the analysis of input data features, although the auto-encoder will still learn the features of the raw data. From the above analysis, a sparsity coder can constrain the input data with useful identification information [35]. Let \(a_{j}^{(2)}(x)\) represent the activation of the \(j\)th hidden units by the auto-encoder from the input \(x\), so that the average activation of the \(j\)th unit in the hidden layer is defined as follows [36]:  
\[\hat{\rho}_{j}=\frac{1}{m}\sum_{i=1}^{m}[a_{j}^{(2)}(x^{(i)})] \tag{7}\]  
Averaged over the input set, let \(\hat{\rho}_{j}\) represent the degree of sparsity, where \(\rho\) is a sparsity parameter. Let \(\hat{\rho}_{j}=\rho\), which is a small value close to 0; typically, the value of \(\rho\) is set to 0.05. To satisfy the condition that the activation is near 0, a penalty term is added to optimise the deviation of \(\hat{\rho}_{j}\) from \(\rho\). The optimisation formula is defined as follows:  
\[\sum_{j=1}^{s_{2}}\rho log\frac{\rho}{\hat{\rho}_{j}}+(1-\rho)log\frac{1-\rho} {1-\hat{\rho}_{j}} \tag{8}\]  
in which the number of hidden-layer units is \(s_{2}\) and \(j\) denotes hidden units in the network. The Kullback-Leibler (KL) divergence is written as follows [37]:  
\[\sum_{j=1}^{s_{2}}KL(\rho\parallel\hat{\rho}_{j}) \tag{9}\]  
where the KL divergence between a Bernoulli random variable with a mean of \(\rho\) and \(\hat{\rho}_{j}\) is defined as below:  
Figure 1: Architecture of an auto-encoder and decoder in a deep neural network (DNN).  
\[KL(\rho\parallel\hat{\rho_{j}})=\rho log\frac{\rho}{\hat{\rho_{j}}}+(1-\rho) log\frac{1-\rho}{1-\hat{\rho_{j}}} \tag{10}\]  
If \(\hat{\rho_{j}}\) equals \(\rho\), then the value of Equation (9) is equal to 0, which means that the KL divergence reaches the minimum value and close to 0, so the overall cost function is defined as follows:  
\[J_{sparse}(W,b)=J(W,b)+\beta\sum_{j=1}^{s_{2}}KL(\rho\parallel\hat{\rho_{j}}) \tag{11}\]  
where \(J(w,b)\) is the squared error cost function and defined as:  
\[J_{sparse}(W,b)=\left[\frac{1}{m}\sum_{i=1}^{m}J(w,b;x^{(i)},y^{(i)})\right]+ \frac{\lambda}{2}\sum_{i=1}^{n_{j}-1}\sum_{i=1}^{s_{j}}\sum_{j=1}^{s_{j}+1}(W_{ ji}^{(l)})^{2} \tag{12}\]  
The first term of the function \(J(W,b)\) is an average sum of squared error over \(m\) samples and outputs. The second term is a weight decay term which can decrease the magnitude of each weight in a neural network and can effectively prevent overfitting. From Equation (11), \(\beta\) is a coefficient to adjust the weight of the sparsity penalty term. The term \(\hat{\rho_{j}}\) depends on the neural network parameters \(W\) and \(b\), and it is optimised using a derivative checking method [38]. The stochastic gradient descent (SGD) approach is adopted to optimise \(W\) and \(b\), and can be written as follows:  
\[W_{ij}(l)=W_{ij}(l)-\varepsilon\frac{\partial}{\partial W_{ij}(l)}C_{sparse} (W,b) \tag{13}\]  
\[b_{i}(l)=bi(l)-\varepsilon\frac{\partial}{\partial b_{i}(l)}C_{sparse}(W,b) \tag{14}\]  
in Equations (13) and (14), \(\varepsilon\) is the learning rate; squared errors are obtained by training over examples and calculating the average \(\hat{\rho_{j}}\). The best values of \(W\) and \(b\) are obtained by back propagation with SGD.