# Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks
## 5 Simulation Results and Discussion
### Custom Normalization for CNNMVN and Experiments  
The experimental results in [62; 63] demonstrated the high performance and potential of CNNMVN. On the one hand, it shows a high accuracy rate, but on the other hand, to achieve this, we should use custom normalization for the error-backpropagation and weight adjustment processes. According to the error-backpropagation algorithm described above, the errors of the fully connected layers should be calculated using (8)-(10) and (20), and the errors of the convolutional layers according to (24) and (25). All these formulas have a normalization factor equal to the number of inputs to the layer, regardless of whether it is a convolutional layer or a fully connected one. The analysis of the error backpropagation process showed that the errors decrease at each layer, starting from the output layer. This leads to the conclusion that the neuron weights are also adjusted less starting from the output layer and moving back to the convolutional layers. This is especially noticeable in (20), where the error is backpropagated from the second hidden layer to the first one, and the normalization factor is equal to the size of the flattened feature maps, which is large in most cases. Thus, we decided to adjust the normalization factors for the CNNMVN learning algorithm. First, we decided to remove the normalization factors in (6), where the neurons in the fully connected layer are adjusted. Second, we removed the normalization factor in (20) when the error is backpropagated from the second hidden layer to the first one. Third, we added self-adapting normalization for the convolutional layers. Hence, the error of each convolved pixel is additionally normalized by the absolute value of the current weighted  
Figure 5: Process of the Fourier coefficients pooling: (**a**) the Fourier coefficients of the image; (**b**) Fourier coefficients with circular shift; (**c**) extraction of the frequencies around DC frequency. sum resulting in this pixel. Now, the feature map errors of the last convolutional layer (24) with additional normalization can be found as follows:  
\[\delta_{ij}^{hC}=\frac{1}{Q_{G}\big{|}\big{|}_{ij}^{h,gC}\big{|}}\sum_{n=1}^{N_{ \mathrm{i}}}\delta_{n,1}\left(w_{ijh}^{n,1}\right)^{-1}, \begin{array}{l}i=1, \ldots,a_{g}-r_{g}+1,\\ j=1,\ldots,b_{g}-r_{g}+1,\\ Q_{G}=r_{G}\cdot r_{G}\cdot c_{G},\end{array}  \tag{33}\]  
and the errors of the feature maps in preceding convolutional layers (25) with additional normalization factor are  
\[\delta_{ij}^{l,g-1}=\frac{1}{Q_{g-1}\big{|}\big{|}_{ij}^{h,g-1}\big{|}}\sum_{h= 1}^{H_{g}}\sum_{t=1}^{T_{g}}\delta^{h,g,t}\left(w_{t}^{h,g}\right)^{-1}, \begin{array}{l}i=1,\ldots,a_{g}-r_{g}+1,\\ j=1,\ldots,b_{g}-r_{g}+1,\\ Q_{g}=r_{g}\cdot r_{g}\cdot c_{g},\end{array}  \tag{34}\]  
where \(\big{|}\big{|}_{ij}^{h,g}\big{|}\) is the absolute value of the convolved pixel with coordinates \(i,j\) in the \(h^{th}\) kernel in the \(g^{th}\) convolutional layer (\(h=\overline{1,H_{g}},g=\overline{2,G}\)) obtained by (17). The adjusted normalization was tested using \"one sample\" learning and a single hidden fully connected layer. Each kernel in the convolutional layer had a stride of 1, and the pool stride (if a pooling layer was used) was 2. We also used the modified soft margins algorithm [74], which skips the learning samples if the sample's errors are less than a predefined soft margins threshold, which was equal to \(\pi/18\). The learning lasted 20 epochs. Experimental results for the MNIST dataset are presented in Table 1, and for the Fashion MNIST dataset in Table 2. The notation is as follows: for example, in topology 16C5-P2-h128-o10, parameter 16C5 means a convolutional layer with 16 kernels and a kernel size of 5 \(\times\) 5; P2 stands for the pooling layer with a window size of 2 \(\times\) 2; h128 is the number of neurons in a single hidden layer in the fully connected part of the network; and o10 stands for the number of output neurons. Each of output neurons performs binary classification using the \"One vs. All\" approach.