## 4 Application of the Proposed Training Method to the Rank-1 CNN
### Construction of the 3-D Rank-1 Filters  
We first observe that a 2-D convolution can be seen as shifting inner products, where each component \(y(\textbf{r})\) at position **r** of the output matrix **Y** is computed as the inner product of a 2-D filter **W** and the image patch \(\textbf{X}(\textbf{r})\) centered at **r**:  
\[y(\textbf{r})=<\textbf{W},\textbf{X}(\textbf{r})>. \tag{6}\]  
If **W** is constructed by the outer product of two 1-D vectors **p** and **q**, i.e., \(\textbf{W}=\textbf{p}\otimes\textbf{q}=\textbf{p}\textbf{q}^{T}\), then **W** becomes a 2-D rank-1 filter. In this case, it can be observed that  
\[y(\textbf{r})=<\textbf{W},\textbf{X}(\textbf{r})>=<\textbf{p}\textbf{q}^{T}, \textbf{X}(\textbf{r})>=\textbf{p}^{T}\textbf{X}(\textbf{r})\textbf{q}. \tag{7}\]  
Figure 2: Proposed rank-1 neural network with different network structures in training and testing phases. Here, \(*\) denotes the convolution operator.  
As has been explained in the case of B2 DPCA, since \(\mathbf{p}\) is multiplied to the rows of \(\mathbf{X}(\mathbf{r})\), \(\mathbf{p}\) tries to extract the features from the rows of \(\mathbf{X}(\mathbf{r})\) which can minimize the loss function. That is, \(\mathbf{p}\) searches the rows in all patches \(\mathbf{X}(\mathbf{r}),\forall_{\mathbf{r}}\) for some common features which can reduce the loss function, while \(\mathbf{q}\) looks for the features in the columns of the patches. In analogy to the B2 DPCA, this bilateral projection removes the redundancies among the rows and columns in the 2-D filters.  
In convolutional neural networks, the input \(\mathbf{X}_{3D}\) and the convolutional filter \(\mathbf{W}_{3D}\) are both three dimensional, where the third dimension refers to the depth of the input, i.e., the numer of input channels. In this case, the 3-D rank-1 filter \(\mathbf{W}_{3D}\) is constructed by the outer product of three 1-D vectors \(\mathbf{p}\), \(\mathbf{q}\), and \(\mathbf{t}\),  
\[\mathbf{W}_{3D}=\mathbf{p}\otimes\mathbf{q}\otimes\mathbf{t}, \tag{8}\]  
where the length of \(\mathbf{t}\) is the same as the depth of the input \(\mathbf{X}_{3d}\). In analogy to the B2 DPCA, the 3-D rank-1 filters which are learned by the three dimensional multilateral projection will have less redundancies among the rows, columns, and the channels than the normal 3-D filters in standard CNNs.  
The three dimensional convolution of the 3-D rank-1 filter \(\mathbf{W}_{3D}\) and \(\mathbf{X}_{3D}\) can be expressed by the sum of channel-wise 2-D convolutions. Let \(\mathbf{W}_{3D}^{(i)}\) denote the \(i\)'s 3-D rank-1 filter that results in the \(i\)'s output channel \(\mathbf{Y}^{(i)}\), and \(\mathbf{W}_{2D}^{(i)}\) the 2-D rank-1 filter that relates with \(\mathbf{Y}^{(i)}\), which is constructed by the outer product of two 1-D vectors \(\mathbf{p}_{l}\) and \(\mathbf{q}_{r}\). We construct a rank-1 2-D filter for each output channel \(\mathbf{Y}^{(i)}\),  
\[\begin{array}{ll}\mathbf{W}_{2D}^{(1)}&=\mathbf{p}_{1}\otimes\mathbf{q}_{1},\\ \mathbf{W}_{2D}^{(2)}&=\mathbf{p}_{1}\otimes\mathbf{q}_{2},\\ &\quad\quad\quad\quad\quad\vdots\\ \mathbf{W}_{2D}^{(i)}&=\mathbf{p}_{l}\otimes\mathbf{q}_{r},\\ \mathbf{W}_{2D}^{(i+1)}&=\mathbf{p}_{l}\otimes\mathbf{q}_{r+1},\\ &\quad\quad\quad\quad\quad\vdots\\ \mathbf{W}_{2D}^{(q)}&=\mathbf{p}_{m}\otimes\mathbf{q}_{n}.\end{array} \tag{9}\]  
The total number of 2-D filters is \(q=m\times n\), where \(q\) is the number of output channels. Then, the 3-D rank-1 filters can be constructed by where \(I_{x}\) and \(I_{y}\) are the width and height of the input feature map, respectively, \(C_{out}\) is the number of output channels, and \(f_{x}\) and \(f_{y}\) are the width and height of the filter.