# Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks
## 3 CNNMVN Learning Algorithm and Error Backpropagation
### CNNMVN Error Backpropagation  
#### 3.2.1 Error Backpropagation in the Fully Connected Part  
Since CNNMVN consists of two parts--a convolutional part and a fully connected part--its error-backpropagation algorithm is divided into two steps. The first step is the error backpropagation through the fully connected (MLMVN) part. Therefore, the global and local errors of all neurons in the fully connected part (except the neurons in the first hidden layer) can be found according to (8)-(10). To understand the process of error backpropagation to the first hidden layer and the convolutional layers, we need to modify the error-backpropagation process in MLMVN specifically for the first hidden layer. We need to take into consideration that in CNNMVN, the first hidden layer in itsfully connected part is not the first hidden layer of the entire network because there are convolutional layers preceding it.  
Let us have a fully connected neural network with \(M\) layers, and let each layer contain \(N_{m}\) neurons (\(m=\overline{1,M}\)). Let us examine the feedforward process between the \(m^{th}\) and \(m+1^{st}\) hidden layers. Let \(z_{im}\) be the weighted sum of the \(i^{th}\) neuron in the \(m^{th}\) layer. After the activation function (4) is applied, we obtain the output of this neuron, which becomes a respective input for the neurons in the \(m+1^{st}\) layer:  
\[x_{i}^{m+1}=P(z_{im}). \tag{18}\]  
Along with other inputs, it forms the weighted sums of the neurons in the \(m+1^{st}\) layer:  
\[z_{n,m+1}=w_{0}^{n,m+1}+w_{1}^{n,m+1}x_{1}^{m+1}+\cdots+w_{i}^{n,m+1}x_{i}^{m+ 1}+\cdots+w_{N_{m}}^{n,m+1}x_{N_{m}}^{m+1}, \tag{19}\]  
where \(z_{n,m+1}\) and \(w_{i}^{n,m+1}\) are the weighted sum and the weights of the \(n^{th}\) neuron of the \(m+1^{st}\) layer, respectively (\(m=\overline{1,N_{m+1}},i=\overline{1,N_{m}}\)). According to (10), the error \(\delta_{im}\) of the \(i^{th}\) neuron of \(m^{th}\) hidden layer is the dot product of the errors of all neurons in the \(m+1^{st}\) layer and the \(i^{th}\) weight of all these neurons. It is essential to note that the \(i^{th}\) weights of all the \(m+1^{st}\) layer neurons are responsible for processing the \(i^{th}\) input of these neurons (Figure 1). We would like to slightly modify the error evaluation here to adapt it to this convolutional network. The obtained error should be normalized not by the number of neurons in the preceding (\(m+1^{st}\)) layer plus one, but by the number of inputs of neurons in the \(m^{th}\) layer. This is because we should share the error among these inputs and then adjust the weights of the neurons that produced them. Hence, we can generalize this approach and say that the error of the \(i^{th}\) neuron in the \(m^{th}\) hidden layer is the dot product of the vector containing the errors of all neurons in the \(m+1^{st}\) layer and another vector containing the reciprocal weights responsible for processing the \(i^{th}\) input of these neurons.  
Figure 1: The \(i^{th}\) weights of the \(m+1^{st}\) layer neurons are responsible for processing the output of the \(i^{th}\) neuron in the preceding \(m^{th}\) layer.  
Now we can generalize this rule for the first hidden layer and the convolutional layers. According to the considerations presented above, the errors of the first-hidden-layer neurons can be found using (10), but with a changed normalization factor:  
\[\delta_{i1}=\frac{1}{F}\sum_{n=1}^{N_{2}}\delta_{n,2}\Big{(}w_{i}^{n,2}\Big{)}^{ -1},\;\;i=\overline{1,N_{1}}, \tag{20}\]  
where \(F\) stands for the number of inputs which is equal to the length of flattened feature maps.  
The convergence of the learning algorithm based on this error-backpropagation rule for MLMVN is proven in [38]. The convergence of the learning algorithm modified here can be proven in the same manner. If we assume that the algorithm does not converge for some input/output mapping that can be implemented using a network, we will arrive at a contradiction with the latter.