# Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks
## 3 CNNMVN Learning Algorithm and Error Backpropagation
### CNNMVN Error Backpropagation
#### 3.2.2 Simple CNNMVN with Two Convolutional Layers  
For simplicity, but without loss of generality, let us consider CNNMVN with two convolutional layers, \(M-1\) hidden layers, and one output layer. Any CNNMVN containing more convolutional layers works in the same way. Let the first convolutional layer contain a single kernel and the second one contain \(H\) kernels. The \(m^{th}\) hidden layer contains \(N_{m}\) neurons (\(m=\overline{1,M-1}\)), and the output layer contains \(N_{M}\) neurons. Let an input image be of size (\(a_{1}\times a_{1}\times 1\)). The kernel size in the first convolutional layer is \(r_{1}\times r_{1}\times 1\) (\(r_{1}<a_{1}\)). Thus, we obtain a single feature map created by the first convolutional layer, and its size should be equal to \(a_{1}-r_{1}+1\times a_{1}-r_{1}+1\times 1\). Let the kernel size in the second convolutional layer be equal to the size of the feature map obtained in the first convolutional layer \(r_{2}\times r_{2}\times 1\) (\(r_{2}=a_{1}-r_{1}+1\)). Hence, from the second convolutional layer we obtain \(H\) feature maps, each consisting of a single pixel. These feature maps should be flattened into a vector and proceed to the first hidden layer and then to the output layer. The global and local errors of the output layer should be calculated according to (8) and (9), respectively. The errors of the neurons in the \(2^{nd},\ldots,M-1^{st}\) hidden layers should be found according to (10), and the errors of the first-hidden-layer neurons should be evaluated according to (20). Since each kernel in the feedforward process forms its own feature map and these feature maps are two-dimensional, we should perform feature map flattening before proceeding to the fully connected layer. As a result, we can assume that this flattened vector of outputs is the same as if it was obtained from a layer in the fully connected part (Figure 2). Therefore, to backpropagate the error from the first hidden layer to the last convolutional layer, we should follow the same approach used for error backpropagation to hidden layers in MLMVN. The distinction is that the errors of the corresponding feature map pixels should then be shared among all inputs of the corresponding kernel, and the respective normalization factor should be equal to the size of the kernel. This leads us to the following formula for the last convolutional layer's errors:  
\[\delta^{h,2}=\frac{1}{Q_{2}}\sum_{n=1}^{N_{1}}\delta_{n,1}\Big{(}w_{h}^{n,1} \Big{)}^{-1},\;\;S_{2}=r_{2}\cdot r_{2}\cdot 1, \tag{21}\]  
where \(\delta^{h,2}\) is the error of the feature map (in this case, a single pixel) produced by the \(h^{th}\) kernel of the second convolutional layer (\(h=\overline{1,H}\)); \(\delta_{n,1}\) are the errors of the first hidden layer neurons and \(w_{h}^{n,1}\) are the neurons' weights in the first hidden layer that are responsible for processing the \(h^{th}\) input of these neurons; and \(Q_{2}\) is the normalization factor, which should be equal to the size of the kernel in the second convolutional layer. To backpropagate the error from the last convolutional layer to the preceding convolutional layers, we should follow the same rule: the error of each pixel of the feature map should be formed from all the weights that process these pixels and the corresponding kernel errors. Before proceeding to these error evaluations, it is important to mention that the image pixels at the corners and edges of an image (or a feature map) and the pixels in the middle of an image (or a feature map) are involved in the respective convolution process a different number of times. For example, a pixel in the top left corner is convolved by the \(k^{th}\) kernel only once, while the next one to the right has already been processed twice (Figure 3). Figure 3: Convolution process of the \(4\times 4\) image by the \(2\times 2\) kernel with stride 1. White pixels determine the kernel position, and a yellow pixel is the target pixel.