## 2 Related Works
### Model Compression and Acceleration
#### 2.1.3 Tensor Decomposition  
Tensors are multidimensional arrays or N-way arrays. The array's dimensionality specifies the tensor order or the number of tensor modes. Tensor decomposition represents high-order tensor data through multilinear operation over its factors. Tensor decomposition methods have attracted considerable attention in various fields such as, psychometrics, chemometrics, machine learning, quantum physics, and neuroscience [43; 44]. CANDECOMP/PARAFAC (CP) and Tucker decomposition are the most popular and well-known algorithms to decompose high-order tensors. Both CANDECOMP/PARAFAC (CP) and Tucker decomposition are high-order generalizations of Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) [45].  
CANDECOMP/PARAFAC (CP) decomposition was presented in the deep learning literature as a tool used to compress the model and reduce the floating-point operation required by convolutional layers and fully connected layers [46]. CP decomposition factorizes a tensor into the sum of the rank-one tensors, as shown in Figure 1. Tucker decomposition is also used to compress the model and reduce the needed floating-point operations required by convolutional layers and fully connected layers [47]. Tucker decomposition compresses the data into tensors of small dimensions represented by core tensors, while its factor matrices span the subspace occupied by the fiber of data [48], as shown in Figure 1. CP decomposition produces a compact representation but makes it challenging to find an optimal solution. Tucker decomposition is stable and flexible but suffers from the curse of the dimensionality problem, in which the number increases exponentially toward the tensor order.  
A tensor network is a generalization of tensor decomposition and considered to be an excellent tool for large-scale data. Tensor networks convert high-order tensors into interconnected, low-order tensors. There are different methods for the use of tensor networks, such as Tensor Train (TT) [49], Hierarchical Tucker (HT) [50], and Tensor Ring (TR) [51]. Tensor Train (TT) is the most common algorithm among tensor network algorithms. Tensor Train (TT) decomposition provides a better representation of high-order dimensional tensors and does not suffer from the curse of dimensionality. Tensor Train (TT) decomposition was applied to a deep learning model to obtain more efficient models in [52; 53]. Hierarchical Tucker is a recursive hierarchical construction of Tucker decomposition [50] that accelerates deep neural network models. Tensor Ring (TR) is a generalized form of CP decomposition. Using two-order tensors instead of one-order tensors forms a ring structure by multiplying the first and last tensor. Tensor ring decomposition was used recently to compresses deep neural network models [51]. Tensor Train (TT), Hierarchical Tucker (HT), and Tensor Ring (TR) are shown in Figure 2.  
The choice of tensor decomposition method depends on the specific architecture of the neural network and the desired trade-off between compression ratio and computational complexity. Each method offers unique advantages in terms of representational power, compression efficiency, and ease of optimization.  
Figure 1: Three different methods of tensor decomposition of a three-order tensor.  
In this paper, we leverage these tensor decomposition techniques, particularly CP, Tucker, and Tensor Train decomposition, to reduce the memory footprint and computational cost of our models while maintaining high accuracy.