## 3 Our Approach
### Applying Tensor Decomposition with Different Rank Methods
#### 3.3.1 CP Decomposition  
CP factorizes a tensor into a linear combination of the rank of one tensor [43]. Figure 1 shows a three-order tensor. The formal definition of CP decomposition for the \(N\)th tensor (\(\mathcal{X}\in\mathbb{R}^{n_{1}\times n_{2}\times\cdots\times n_{d}}\)) decomposed in the outer product matrices and \(R\) is the rank of \(\mathbf{U}_{1},\mathbf{U}_{2},\ldots\mathbf{U}_{d}\), as follows:\[\mathcal{X}=\sum_{r=1}^{R}\mathbf{u}_{\mathbf{1}_{r}}\circ\mathbf{u}_{\mathbf{2}_{ r}}\circ\ldots\mathbf{u}_{\mathbf{d}_{r}} \tag{1}\]  
The factor matrices are the combination of the vectors from the rank-one components, such as  
\(\mathbf{u}=\left[ \begin{array}{cc}\mathbf{u}_{\mathbf{1}}&\mathbf{u}_{ \mathbf{1}_{2}}&\cdots&\mathbf{u}_{\mathbf{1}_{R}}\end{array} \right]\)--and likewise for \(\mathbf{U}_{\mathbf{2}}\) and \(\ldots\mathbf{U}_{\mathbf{d}}\). Columns \(\mathbf{U}_{1}\), \(\mathbf{U}_{2}\), and \(\ldots\mathbf{U}_{\mathbf{d}}\) are very often normalized to the unit length, with weights absorbed into a vector (\(\lambda\in\mathbb{R}^{R}\)) as follows:  
\[\mathcal{X}=\sum_{r=1}^{R}\lambda_{r}\mathbf{u}_{\mathbf{1}_{r}}\circ\mathbf{ u}_{\mathbf{2}_{r}}\circ\ldots\mathbf{u}_{\mathbf{d}_{r}} \tag{2}\]  
For a given tensor, there are several algorithms available to compute CP decomposition. In this paper, we employ an alternative least squares (ALS) Algorithm 2, the core idea of which is to individually optimize for each factor matrix, keeping all tensor factor matrices fixed, except the one that is optimized, and repeating this task for each matrix until the stopping criterion is satisfied [43]. ```
0: Data tensor \(\mathcal{X}\in\mathbb{R}^{n_{1}\times n_{2}\times\cdots\times n_{d}}\) and rank \(R\). 0: Factor matrices \(\mathbf{U}_{1}\in\mathbb{R}^{n_{1}\times R}\), \(\mathbf{U}_{2}\in\mathbb{R}^{n_{2}\times R},\ldots,\mathbf{U}_{d}\in\mathbb{R }^{n_{d}\times R}\). 1:procedure ALS-CP(\(\mathcal{X}\), \(R\))
2: Initialize \(\mathbf{U}_{1},\mathbf{U}_{2},\ldots,\mathbf{U}_{d}\). 3:while not converged or criterion not satisfied do
4:\(\mathbf{U}_{1}\leftarrow\mathcal{X}_{(1)}(\mathbf{U}_{d}\odot\mathbf{U}_{d- 1}\odot\cdots\odot\mathbf{U}_{2})\Big{(}\mathbf{U}_{d}^{\mathrm{T}}\mathbf{U} _{d}\oplus\mathbf{U}_{d-1}^{\mathrm{T}}\mathbf{U}_{d-1}\rightsquigarrow 0.07556 pt\cdots \rightsquigarrow 0.07556 pt\mathbf{U}_{2}^{\mathrm{T}}\mathbf{U}_{2}\Big{)}^{-1}\)
5: Normalize the columns of \(\mathbf{U}_{1}\) to unit length. 6:\(\mathbf{U}_{2}\leftarrow\mathcal{X}_{(2)}(\mathbf{U}_{d}\odot\mathbf{U}_{d- 1}\odot\cdots\odot\mathbf{U}_{1})\Big{(}\mathbf{U}_{d}^{\mathrm{T}}\mathbf{U} _{d}\oplus\mathbf{U}_{d-1}^{\mathrm{T}}\mathbf{U}_{d-1}\rightsquigarrow 0.07556 pt \cdots\rightsquigarrow 0.07556 pt\mathbf{U}_{1}^{\mathrm{T}}\mathbf{U}_{1}\Big{)}^{-1}\)
7: Normalize the columns of \(\mathbf{U}_{2}\) to unit length. 8:\(\vdots\)
9:\(\mathbf{U}_{d}\leftarrow\mathcal{X}_{(d)}(\mathbf{U}_{d-1}\odot\mathbf{U}_{d- 2}\odot\cdots\odot\mathbf{U}_{1})\Big{(}\mathbf{U}_{d-1}^{\mathrm{T}}\mathbf{U }_{d-1}\rightsquigarrow 0.07556 pt\mathbf{U}_{d-2}\rightsquigarrow 0.07556 pt\cdots \rightsquigarrow 0.07556 pt\mathbf{U}_{1}^{\mathrm{T}}\mathbf{U}_{1}\Big{)}^{-1}\)
10: Normalize the columns of \(\mathbf{U}_{d}\) to unit length. 11:endwhile
12: Store the norms in vector \(\lambda\). 13:return\(\mathbf{U}_{1},\mathbf{U}_{2},\ldots,\mathbf{U}_{d}\) and \(\lambda\). 14:endprocedure
```  
**Algorithm 2** ALS for CP decomposition [44]  
We use CP tensor decomposition on the convolutional and fully connected layers. The rank of the tensor is required to apply the decomposition. Finding the tensor's rank is an NP-hard problem.There are numerous algorithms and methods available to approximate the tensor rank. In this paper, we implement and explore the three previously explained approaches to select the ranks. First, we use a random rank for all of the layers, which is a random number based on the size of the tensor. In the second approach, we select the rank based on the layer sensitivity for decomposition.