# Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks
## 2 MVN and MLMVN Fundamentals
### MLMVN  
As mentioned earlier, MLMVN is a fully connected feedforward neural network based on MVN. There are two types of learning algorithms for MLMVN. The first one is the \"one sample learning\" or serial learning, where the network learns only one sample at each iteration [38; 60; 61]. The second one is the batch learning algorithm [74; 75] where the network can adjust its weights based on the errors of multiple samples simultaneously. This approach speeds up the learning process and improves the generalization capability of the network. Let us briefly recall these learning rules for MLMVN in the most general case. We will employ them to design the CNNMVN learning algorithm. Let us use the following notation: in general, MLMVN has \(M\) layers: \(M-1\) hidden layers, and one output layer. Let each layer contain \(N_{m}\) neurons \(m=\overline{1,M}\). Let \(D_{nM}\), where \(n=\overline{1,N_{m}}\) be the desired output, and \(Y_{nM}\) be the actual output of the \(n^{th}\) neuron in the output layer. Then the global error of the network should be calculated as  
\[\delta^{*}_{nM}=D_{nM}-Y_{nM},\;\;n=\overline{1,N_{M}}. \tag{8}\]  
According to an error-sharing principle [38], we assume that the global error \(\delta^{*}_{nM}\) accumulates the local errors from all neurons contributed to corresponding output, and therefore we need to share this global error among all these neurons. Then the local errors \(\delta_{nM}\) of the output neurons are  
\[\delta_{nM}=\frac{1}{N_{M-1}+1}\delta^{*}_{nM},\;\;n=\overline{1,N_{M}}. \tag{9}\]  
Now these errors should be backpropagated to the neurons in the hidden layers:  
\[\delta_{im}=\frac{1}{q_{m-1}}\sum_{n=1}^{N_{m+1}}\delta_{n,m+1}\Big{(}w_{i}^{ n,m+1}\Big{)}^{-1},\;\;i=\overline{1,N_{m}},\;m=\overline{1,M-1}, \tag{10}\]where \(\delta_{im}\) is the error of the \(i^{th}\) neuron in the \(m^{th}\) layer, and the normalization factor \(q_{m-1}=N_{m-1}+1\left(m=\overline{2,M-1}\right)\) is the number of all neurons in the \(m-1^{st}\) layer plus one (note that for the first layer (\(m=1\)) we use \(q_{1}=1\) because there are no preceding layers and no neurons to share the error with). Now the Equations (8)-(10) determine the error-backpropagation rule for MLMVN, and the neurons' weights can be adjusted for standard \"one sample learning\" according to (6) or (7). To apply the batch technique, we need to evaluate all the errors according to (8)-(10) for all samples from the batch. For simplicity, but without loss of generality, let us consider an arbitrary neuron in the network with \(p\) inputs and \(p+1\) weights. Let the learning set contain \(S\) samples where the \(s^{th}\) neuron's input vector is \(X_{s}=\left(1,\ x_{s1},\ldots,x_{sp}\right),s=\overline{1,S}\). Let the adjustment terms for the neurons' weights be \(\left(\Delta w_{0},\ldots,\Delta w_{p}\right)\), and the errors for each sample be \(\left(\delta_{1},\ldots,\delta_{S}\right)\). It was shown in [75] that the adjustments for the neurons' weights can be represented as  
\[ \begin{bmatrix}1&x_{11}&\cdots&x_{1p}\\ \vdots&\vdots&&\vdots\\ 1&x_{S1}&\cdots&x_{sp}\end{bmatrix}\begin{bmatrix}\Delta w_{0}\\ \vdots\\ \Delta w_{p}\end{bmatrix} = \begin{bmatrix}\delta_{1}\\ \vdots\\ \delta_{S}\end{bmatrix} , \tag{11}\]  
or simply  
\[X\cdot\Delta w=\delta. \tag{12}\]  
This system of linear algebraic equations for unknowns \(\left(\Delta w_{0},\ldots,\Delta w_{p}\right)\) is typically overdetermined \(\left(S\gg p+1\right)\), and we can find a unique least squares solution as  
\[\Delta w=X^{\ddagger}\cdot\delta, \tag{13}\]  
where \(X^{\ddagger}\) is the Moore-Penrose pseudoinverse of \(X\), which is the pseudoinverse matrix of \(X\). Now the adjusted weights of the hidden neuron are:  
\[\widetilde{W}=W+C\cdot\Delta w, \tag{14}\]  
where \(W\) and \(\widetilde{W}\) are the weighting vectors before and after correction, \"+\" denotes component-wise addition, and \(C\) stands for the learning rate.