## 5 Application of the Proposed Training Method to the MobileNet  
Here, we show that the proposed rank-1 network training method can also be applied to train the well-known MobileNet (version 1). However, the performance becomes better when the parameters are obtained by the proposed method, than when obtained by the original MobileNet type training method. The main idea of applying the proposed training method to the training of the MobileNet is that we can extend the separate 2-D filters to 3-D filters by the outer product of rank-1 2-D filters with rank-1 1-D vectors resulting in rank-1 3-D filters, train the rank-1 3-D filters with the ATCD training method, and then compress the rank-1 3-D filters back to full-rank 2-D filters. In the original version (version 1) of the MobileNet, the 2-D images are separately convolved with 2-D filters, and then are combined by \(1\times 1\) convolutions. The output of a single layer of the original version of the MobileNet can be written as  
\[\mathbf{Y}^{(m)}=\sum_{j=1}^{N}\mathbf{a}_{m}[j](\mathbf{X}^{(j)}\oplus\mathbf{W}^{(j)}),m=1,...,q, \tag{25}\]  
where \(\mathbf{Y}^{(m)}\) is the \(m\)'s output channel, \(\mathbf{X}^{(j)}\) is the \(j\)'s input channel, \(\mathbf{W}^{(j)}\) is the \(j\)'s filter that convolves with the \(j\)'s input channel, \(\oplus\) is the 2-D convolution operator, and \(\mathbf{a}_{m}\) is the \(m\)'s \(1\times 1\) convolution filter that produces the \(m\)'s output channel. Meanwhile, the outputs \(\hat{\mathbf{Y}}^{(i)}\), \(i=1,...,K\) which are obtained by the convolutions of the 3-D rank-1 filters and the input channels as shown in Figure 3 become  
\[\hat{\mathbf{Y}}^{(i)}=\sum_{j=1}^{N}\mathbf{t}_{i}[j](\mathbf{X}^{(j)}\oplus\hat{\mathbf{ W}}^{(i)}),i=1,...,K. \tag{26}\]  
It has to be noted that the index of \(\hat{\mathbf{W}}^{(i)}\) in (26) is \(i\) (the index of output channels) compared to (25) where the index of \(\mathbf{W}^{(j)}\) is \(j\) (the index of input channels). Now, adding an extra layer which computes the linear combinations of the outputs \(\hat{\mathbf{Y}}^{(i)},i=1,...,K\) in (26) by \(1\times 1\) convolutions with the filters \(\mathbf{k}_{m},m=1,...,q\), we have  
\[\mathbf{Y}^{(m)}=\sum_{i=1}^{K}\mathbf{k}_{m}[i]\hat{\mathbf{Y}}^{(i)},m=1,...,q. \tag{27}\]  
By putting (26) into (27) and rearranging the order of summation, we get,  
\[\mathbf{Y}^{(m)}=\sum_{j=1}^{N}\Bigg{[}\mathbf{X}^{(j)}\oplus\Bigg{(}\sum_{i=1}^{K} \mathbf{k}_{m}[i]\mathbf{t}_{i}[j]\hat{\mathbf{W}}^{(i)}\Bigg{)}\Bigg{]}. \tag{28}\]After the values of \(\mathbf{k}_{m}[i]\) and \(\mathbf{t}_{i}[j]\) are all fixed for all \(m\), \(i\), and \(j\), i.e., after they have been trained, we can arbitrarily construct the vectors \(\mathbf{a}_{m,\,m=1,\ldots,q}\) and \(\mathbf{b}_{j,\,j=1,\ldots,N}\) so that the entries in the vectors are assigned as follows:  
\[\mathbf{a}_{m}[j]=\mathbf{k}_{m}[i],\ \mathbf{b}_{j}[i]=\mathbf{t}_{i}[j]. \tag{29}\]  
Then, we can rewrite (28) as  
\[\mathbf{Y}^{(m)}=\sum_{j=1}^{N}\Bigg{[}\mathbf{X}^{(j)}\otimes\left(\sum_{i=1 }^{K}\mathbf{a}_{m}[j]\mathbf{b}_{j}[i]\hat{\mathbf{W}}^{(i)}\right)\Bigg{]}, \tag{30}\]  
which can now be rewritten as  
\[\mathbf{Y}^{(m)}=\sum_{j=1}^{N}\mathbf{a}_{m}[j]\Bigg{[}\mathbf{X}^{(j)} \otimes\left(\sum_{i=1}^{K}\mathbf{b}_{j}[i]\hat{\mathbf{W}}^{(i)}\right) \Bigg{]}. \tag{31}\]  
By letting  
\[\mathbf{W}^{(j)}=\sum_{i=1}^{K}\mathbf{b}_{j}[i]\hat{\mathbf{W}}^{(i)}, \tag{32}\]  
the formula becomes the same as that for the Mobilenet described in (25). This means that after being training by the proposed method, we can implement the inference system also in the Mobilenet style. It has to be noticed that even though \(\hat{\mathbf{W}}^{(i)}\) is a two dimensional rank-1 filter, since it is composed of the outer product of two 1-D vectors, the filter \(\mathbf{W}^{(j)}\) can have a rank of \(K\), as the summation of \(K\) independent rank-1 filters results in a rank-\(K\) filter.