# Frequency-Domain and Spatial-Domain MLMVN-Based Convolutional Neural Networks
## 3 CNNMVN Learning Algorithm and Error Backpropagation
### Adjustment of the Weights in CNNMVN  
After the error backpropagation is complete, all the weights in both the convolutional and fully connected parts of CNNMVN should be adjusted. To design this process for CNNMVN, we should base it on the respective process of weight adjustment in MLMVN.  
A kernel in any convolutional layer basically works similarly to MVN--it calculates a weighted sum of inputs and applies an activation function to it. Hence, to adjust the kernels' weights, we can employ the same idea as in the error-correction learning rules (6) and (7), which are used to adjust the weights in MVN and MLMVN. However, this idea requires some adaptation. There is one significant distinction between MVN and a kernel in a convolutional layer. An MVN processes one vectorized input at a time and, based on this input, it produces a single output. However, a kernel in any convolutional layer slides over an image (or a feature map), processing multiple inputs and producing multiple outputs accordingly. This means that the error-correction learning rules (6) and (7) for MVN and MLMVN cannot be used directly for CNNMVN. However, in [62], we proposed to employ for kernels' weight adjustment the same idea of the batch LLS-based learning algorithm [75], which was developed for MLMVN to simultaneously adjust the weights for multiple learning samples belonging to a batch.  
Let us have an \(a\times b\times d\) input image and a kernel whose size is \(r\times r\times d\) (\(r<\min(a,b)\)). Let \(K\) be the number of the convolutional windows in an image through which a convolutional kernel is sliding. Thus \(K=(a-r+1)\times(b-r+1)\). Each convolutional window can be flattened and represented as an input vector to a kernel. Then, using a matrix-vector notation, the convolutional operation can be represented as  
\[\begin{bmatrix}x_{1}\\ \vdots\\ x_{K}\end{bmatrix}\begin{bmatrix}w_{1,1,1}\\ \vdots\\ w_{r,d}\end{bmatrix}=\begin{bmatrix}z_{1}\\ \vdots\\ z_{K}\end{bmatrix}, \tag{27}\]  
where \(x_{i}\) is the flattened input vector (\(i=\overline{1,K}\)), \(w\) are the kernel weights, and \(z_{i}\) are the convolved pixels. According to a batch algorithm and as it was shown in [62], after calculating the errors for all \(z_{i}\), we can represent adjustments, which should be added to the weights to correct them similarly to (11) and obtain the following:  
\[\begin{bmatrix}x_{1}\\ \vdots\\ x_{K}\end{bmatrix}\begin{bmatrix}\Delta w_{1,1,1}\\ \vdots\\ \Delta w_{r,r,d}\end{bmatrix}=\begin{bmatrix}\delta_{1}\\ \vdots\\ \delta_{K}\end{bmatrix}, \tag{28}\]  
where \(\Delta w_{u,v,j}\) are the adjustments terms for kernel weights (\(u,v=\overline{1,r}\) and \(j=\overline{1,d}\)) and \(\delta_{i}\) are the respective errors to be corrected (\(i=\overline{1,K}\)). We can rewrite (28) as  
\[X\cdot\Delta w=\delta, \tag{29}\]  
where \(X\) consists of \(K\) flattened inputs. This is similar to the MLMVN batch learning described by (11) and (12), where the neural network weights are adjusted simultaneously for multiple learning samples belonging to the same batch. Thus, the system of Equations (28) or (29), which is the same) for unknowns (\(\Delta w_{1,1,1},\ldots,\Delta w_{r,d}\)), which is typically overdetermined (\(K\gg r\cdot r\cdot d\)), can be solved in a similar way:  
\[\Delta w=X^{\dagger}\cdot\delta, \tag{30}\]where \(X^{\ddagger}\) is the Moore-Penrose inversion of \(X\), that is a pseudoinverse matrix of \(X\). Finally, the adjusted weights of the kernel are  
\[\widetilde{W}=W+C\cdot\Delta w, \tag{31}\]  
where \(W\) and \(\widetilde{W}\) are the weighting vectors before and after correction, \"+\" is a component-wise addition, and \(C\) stands for learning rate. Since the kernel has multiple inputs and outputs, the batch approach allows finding the \"best fit\" adjustments for all inputs.