## 2 Methodology
### Feature Compression
#### 2.3.2 Sparse Auto-Encoder with Non-Negativity Constraint  
A sparse auto-encoder (\(SAE\)) results from an auto-encoder (\(AE\)). Based on \(AE\), \(SAE\) neural network is achieved by enforcing a sparsity constraint of the output from the hidden layer, which realizes the inhibitory effects and yields fast convergence speed for training process using the back propagation algorithm [17; 55]. Hence, the cost function of \(SAE\) is given by  
\[E_{SAE}=J_{mse}+\lambda\Omega_{w}+\beta\Omega_{s}, \tag{16}\]  
where \(\beta\) is the coefficient of the sparsity regularization term, \(\Omega_{s}\) is the sparsity regularization term which is usually represented by Kullback-Leibler (KL) divergence [17; 55]. The part-based representation of input data usually exhibits excellent performance for pattern classification. The sparse representation scheme usually breaks the input data into parts, while the original input data can be readily reconstructed by combining the parts additively when necessary. Therefore, the input in each layer of an auto-encoder can be divided into parts by enforcing the weight coefficients of both encoder and decoder to be positive [56]. To achieve a better performance in reconstruction, we propose to consider the sparse auto-encoder with non-negativity constraint (NC-SAE), the auto encoder network decompose the input into parts by encoder via (8 and 9), and combine them in an additive manner by decoder via (11 and 12). This is achieved by replacing the regularization term (15) in cost function (16) with a new non-negativity constraint  
\[\Phi_{w}=\frac{1}{2}\sum_{l=1}^{L}\sum_{j=1}^{m}\sum_{i=1}^{n}{(\phi(w_{ji}^{(l )}))}, \tag{17}\]  
where  
\[\phi(w_{ji}^{(l)})=\left\{ \begin{array}{cc}{(w_{ji}^{(l)})}^{2}&w_{ji}<0\\ 0&w_{ji}\geq 0\end{array} \right.. \tag{18}\]  
Therefore, the proposed cost function for NC-SAE is defined as  
\[E=J_{mse}+\alpha\Phi_{w}+\beta\Omega_{s}, \tag{19}\]  
where \(\alpha\geq 0\) is the parameter of the non-negativity constraint. By minimizing the cost function (19), the number of nonnegative weights of each layer and the sparsity of the hidden layer activation are all increased, and the overall average reconstruction error is reduced. Further, steepest descent method is used to update the weight and bias of (19) as follows  
\[ \begin{array}{l}w(k+1)=w(k)-\eta\frac{\partial E}{\partial w(k)}\\ b(k+1)=b(k)-\eta\frac{\partial E}{\partial b(k)}\end{array} , \tag{20}\]  
where \(k\) is the number of iteration, and \(\eta\) denotes the learning rate. Then, we adopt the error back-propagation algorithm to compute the partial derivatives in (20). The partial derivatives of the cost function with respect to decoder reads  
\[\frac{\partial E}{\partial w_{ij}^{(2)}}=\frac{\partial J_{mse}}{\partial w_{ ij}^{(2)}}+\alpha\frac{\partial\Phi_{w}}{\partial w_{ij}^{(2)}}+\beta\frac{ \partial\Omega_{\rm s}}{\partial w_{ij}^{(2)}}, \tag{21}\]  
The partial derivatives in (21) are straightforward, and shown below  
\[\frac{\partial J_{mse}}{\partial w_{ij}^{(2)}}=\frac{\partial J_{mse}}{ \partial a_{i}^{(2)}}\frac{\partial a_{i}^{(2)}}{\partial w_{ij}^{(2)}}=\frac {\partial J_{mse}}{\partial a_{i}^{(2)}}y_{j}, \tag{22a}\] \[\frac{\partial\Phi_{w}}{\partial w_{ij}^{(2)}}=r(w_{ij}^{(2)}),\] (22b) \[\frac{\partial\Omega_{\rm s}}{\partial w_{ij}^{(2)}}=0, \tag{22c}\]  
where \(r(w)\) is shown as follows  
\[r(w_{ij}^{(l)})=\left\{ \begin{array}{c}w_{ij}^{(l)},w_{ij}^{(l)}<0\\ 0,\text{otherwise}\end{array} \right.. \tag{23}\]  
In order to clarify the computation of derivatives, we define the neuronal error \(\delta\) as the derivative of cost function with respect to weight input of each neuron, i.e., \(\delta\triangleq\partial E/\partial a\).