## 3 Convolutional Neural Network
### Principle and Proposed Structure  
#### 3.1.1 Principle  
(1) Convolution  
In the convolution layer, the feature map from the upper layer is convoluted by the convolution kernel, and then the output feature map is obtained via the activation function [33]:  
\[x_{j}^{\ell}=f(u_{j}^{\ell}), \tag{1}\]  
\[u_{j}^{\ell}=\sum_{i\in M_{j}}x_{i}^{\ell-1}*k_{ij}^{\ell}+b_{j}^{\ell}, \tag{2}\]  
where \(u_{j}^{\ell}\) is the net activation of the \(j\) channel in the convolution layer, which is obtained by summing the convolution and bias of the output feature map \(x_{i}^{\ell-1}\) of the upper layer. \(x_{j}^{\ell}\) is the output of the \(j\) channel of the convolution layer. \(f\) (\(\cdot\)) is called the activation function, and it is a ReLU function in this paper. \(M_{j}\) represents a subset of input feature maps for computing, \(k_{ij}^{\ell}\) is a convolution kernel, and \(b_{j}^{\ell}\) is the bias item of the feature map after convoluting. For an output feature map \(x_{j}^{\ell}\), the convolution kernel \(k_{ij}^{\ell}\) corresponding to each input feature map \(x_{i}^{\ell-1}\) may be different. (2) Pooling  
The output feature map is obtained by the down sampling layer by sampling every input feature map by the following formula:  
\[x_{j}^{\ell}=f(\beta_{j}^{\ell}\text{down}(x_{j}^{\ell-1})+b_{j}^{\ell}), \tag{3}\]  
where \(\beta_{j}^{\ell}\) is the weight coefficient of the down sampling layer, and \(b_{j}^{\ell}\) is the bias of the down sampling layer. The symbol down(\(\cdot\)) represents the down sampling function, which calculates the sum, mean or maximum value of the pixel in the \(n\times n\) region of the input feature map so that the output map is reduced by \(n\) times in two dimensions. (3) Full connection  
In the fully connected network, all two-dimensional image features are stitched into one-dimensional features as inputs to the fully connected network. The output of the full connection layer can be obtained by weighting and by the activation function:  
\[x^{\ell}=f(u^{\ell}), \tag{4}\]  
\[u^{\ell}=w^{\ell}x^{\ell-1}+b^{\ell}, \tag{5}\]  
where \(w^{\ell}\) is the weight coefficient of the fully connected network, and \(b^{\ell}\) is the bias item of the fully connected layer. (4) Classification  
To solve the multi-classification problem, the Softmax [34] function, which is located in the last layer, is usually used. It is expressed as the probabilistic expression \(p(y=j/x)\), where \(x\) is the input sample and the corresponding label is \(y\), \(p\) is the probability of sample \(j\). Therefore, the output will be an \(n\)-dimensional vector for a classifier with \(n\) classes and the sum of the elements in a vector is 1, as shown by Equation (6) [20; 35]:\[p(y^{(i)}=n\Big{|}x^{(i)};w)=\left[ \begin{array}{c}p(y^{(i)}=1\Big{|}x^{(i)} \,;w)\\ p(y^{(i)}=2\Big{|}x^{(i)}\,;w)\\ \vdots\\ p(y^{(i)}=n\Big{|}x^{(i)}\,;w)\end{array} \right]=\frac{1}{\sum_{j=1}^{n}e^{y^{ \mathrm{T}}x^{(i)}}}\left[ \begin{array}{c}e^{w^{\mathrm{T}}x^{(i)}}\\ e^{w^{\mathrm{T}}x^{(i)}}\\ \vdots\\ e^{w^{\mathrm{T}}_{n}x^{(i)}}\end{array} \right], \tag{6}\]  
where \(w\) is the weight, and \(w^{\mathrm{T}}_{n}x^{(i)}\) are the inputs of the Softmax layer. The term \(1/\sum_{j=1}^{n}e^{w^{\mathrm{T}}_{j}x^{(i)}}\) normalizes the distribution, so that it sums to 1 [20]. In the training process, the optimization algorithm is used to minimize the loss function to complete the network training. The loss function \(J(\theta)\) is defined by Equation (7) [35]:  
\[J(\theta)=-\frac{1}{m}\left[\sum_{i=1}^{m}\sum_{j=1}^{n}1\Big{\{}y^{(i)}=j \Big{\}}\log\frac{e^{w^{\mathrm{T}}_{j}x^{(i)}}}{\sum_{l=1}^{n}e^{w^{\mathrm{ T}}_{l}x^{(i)}}}\right], \tag{7}\]  
where \(1\Big{\{}y^{(i)}=j\Big{\}}\) is an indicator function that always returns 1 or 0, which means that when a predicted class of the \(i\)th input is true for class \(j\), the result is 1; otherwise, the result is 0.