## 4 Application of the Proposed Training Method to the Rank-1 CNN
### Training Process  
Figure 4 explains the training process with the proposed network structure in detail. At every epoch of the training phase, we first take the outer product of the three 1-D vectors \(\mathbf{p}\), \(\mathbf{q}\), and \(\mathbf{t}\). Then, we assign the result of the outer product to the weight values of the 3-D convolution filter, i.e., for every weight value in the 3-D convolution filter \(\mathbf{W}\), we assign  
\[w_{i,j,k}=p_{i}q_{j}t_{k},\ \forall_{i,j,k\in\Omega(\mathbf{W})} \tag{14}\]  
where, \(i,j,k\) correspond to the 3-D coordinates in \(\Omega(\mathbf{W})\), the 3-D domain of the 3-D convolution filter \(\mathbf{W}\). Since the matrix constructed by the outer product of vectors has always a rank of one, the 3-D convolution filter \(\mathbf{W}\) is a rank-1 filter. During the back-propagation phase, every weight value in \(\mathbf{W}\) will be updated by  
\[w^{\prime}_{i,j,k}=w_{i,j,k}-\alpha\frac{\partial L}{\partial w_{i,j,k}}, \tag{15}\]  
where \(\frac{\partial L}{\partial w_{i,j,k}}\) denotes the gradient of the loss function \(L\) with respect to the weight \(w_{i,j,k}\), and \(\alpha\) is the learning rate. In standard convolutional neural networks, \(w^{\prime}_{i,j,k}\) in (15) is the final updated weight value at each update step. However, the updated filter \(\mathbf{W}^{\prime}\) normally is not a rank-1 filter. This is due to the fact that the update in (15) is done in the direction which considers only the minimizing of the loss function and not the rank of the filter. With the proposed training network structure, we take a further update step, i.e., we update the 1-D vectors \(\mathbf{p}\), \(\mathbf{q}\), and \(\mathbf{t}\):  
\[p^{\prime}_{i}=p_{i}-\alpha\frac{\partial L}{\partial p_{i}},\ \forall_{i\in\Omega(\mathbf{p})} \tag{16}\]\[q_{j}^{\prime}=q_{j}-\alpha\frac{\partial L}{\partial q_{j}},\ \forall_{j\in\Omega(\mathbf{q})} \tag{17}\]  
\[t_{k}^{\prime}=t_{k}-\alpha\frac{\partial L}{\partial t_{k}},\ \forall_{k\in\Omega(\mathbf{t})} \tag{18}\]  
Here, \(\frac{\partial L}{\partial p_{i}},\frac{\partial L}{\partial q_{j}}\), and \(\frac{\partial L}{\partial t_{k}}\) can be calculated as  
\[\frac{\partial L}{\partial p_{i}}=\sum_{j}\sum_{k}\frac{\partial L}{\partial w _{i,j,k}}\frac{\partial w_{i,j,k}}{\partial p_{i}}=\sum_{j}\sum_{k}\frac{ \partial L}{\partial w_{i,j,k}}q_{j}t_{k}, \tag{19}\]  
\[\frac{\partial L}{\partial q_{j}}=\sum_{i}\sum_{k}\frac{\partial L}{\partial w _{i,j,k}}\frac{\partial w_{i,j,k}}{\partial q_{j}}=\sum_{i}\sum_{k}\frac{ \partial L}{\partial w_{i,j,k}}p_{i}t_{k}, \tag{20}\]  
\[\frac{\partial L}{\partial t_{k}}=\sum_{i}\sum_{j}\frac{\partial L}{\partial w _{i,j,k}}\frac{\partial w_{i,j,k}}{\partial t_{k}}=\sum_{i}\sum_{j}\frac{ \partial L}{\partial w_{i,j,k}}p_{i}q_{j}. \tag{21}\]  
At the next feed forward step in the back-propagation, an outer product of the updated 1-D vectors \(\mathbf{p}\), \(\mathbf{q}\), and \(\mathbf{t}\) is taken to concatenate them back into a 3-D convolution filter \(\mathbf{W}^{\prime\prime}\), which we call the tensor composing step:  
\[w_{i,j,k}^{\prime\prime}=p_{i}^{\prime}q_{j}^{\prime}t_{k}^{\prime}=(p_{i}- \alpha\frac{\partial L}{\partial p_{i}})(q_{j}-\alpha\frac{\partial L}{ \partial q_{j}})(t_{k}-\alpha\frac{\partial L}{\partial t_{k}})\]  
\[=p_{i}q_{j}t_{k}-\alpha(p_{i}q_{j}\frac{\partial L}{\partial t_{k}}+q_{j}t_{k} \frac{\partial L}{\partial p_{i}}+p_{i}t_{k}\frac{\partial L}{\partial q_{j}} )+\alpha^{2}(p_{i}\frac{\partial L}{\partial q_{j}}\frac{\partial L}{\partial t _{k}}+q_{j}\frac{\partial L}{\partial p_{i}}\frac{\partial L}{\partial t_{k}} +t_{k}\frac{\partial L}{\partial p_{i}}\frac{\partial L}{\partial t_{k}})- \alpha^{3}\frac{\partial L}{\partial p_{i}}\frac{\partial L}{\partial q_{j}} \frac{\partial L}{\partial t_{k}} \tag{22}\]  
\[w^{\prime}_{i,j,k}=w_{i,j,k}-\alpha\frac{\partial L}{\partial w_{i,j,k}}, \tag{15}\]0  
\[w^{\prime}_{i,j,k}=w_{i,j,k}-\alpha\frac{\partial L}{\partial w_{i,j,k}}, \tag{15}\]1 Steps in the training phase of the proposed rank-1 network.