# Decomposition Methods for Machine Learning with Small, Incomplete or Noisy Datasets
## 2 Methods
### Tensor Decomposition (TD)  
Sometimes input data samples have a multidimensional structure or it is useful to arrange one-dimensional signals into multidimensional arrays or tensors. For example, EEG signals are simultaneously recorded with multiple sensors (electrodes) thus, for each subject, a (time \(\times\) channel) matrix is recorded. A natural way to construct a tensor for an EEG experiment is to use a third dimension to index subject, which results in a three dimensional data tensor \(\underline{\mathbf{X}}\in\mathds{R}^{I_{1}\times I_{2}\times I_{3}}\), where \(I_{1},I_{2},I_{3}\) correspond to numbers of time samples, channels (sensors) and subjects, respectively.  
Matrix factorization models, such as PCA in Equation (3), can be generalized to tensors by means of the Tucker(\(R_{1},R_{2},R_{3}\)) decomposition [46]. Given a data tensor \(\underline{\mathbf{X}}\in\mathds{R}^{I_{1}\times I_{2}\times I_{3}}\), it can be decomposed as:  
\[\underline{\mathbf{X}}=\underline{\mathbf{G}}\times_{1}\mathbf{A}_{1}\times_ {2}\mathbf{A}_{2}\times_{3}\mathbf{A}_{3}+\underline{\mathbf{B}}, \tag{6}\]  
where \(\times_{n}\) is the mode-\(n\) tensor-by-matrix product. \(\underline{\mathbf{G}}\in\mathds{R}^{R_{1}\times R_{2}\times R_{3}}\) is the core tensor and \(\mathbf{A}_{n}\in\mathds{R}^{I_{n}\times R_{n}}\) are factor matrices (Figure 1e). As a particular case, when core tensor is diagonal with \(R=R_{1}=R_{2}=R_{3}\) this model is reduced to the CANDECOMP/PARAFAC or CP(\(R\)) decomposition model (see Figure 1f), which has demonstrated to be very useful in a wide range of applications [47; 48]. It is interesting to note that Equation (6) can be also written as a sum of rank-\(1\) tensors as shown in Figure 1h which, if vectorized, it is then reduced to our general decomposition model of Equation (2) as follows:  
\[\mathbf{x}=\sum_{i_{1},i_{2},i_{3}}g_{i_{1}i_{2}i_{3}}\boldsymbol{\phi}_{i_{ 1}i_{2}i_{3}}+\mathbf{r}, \tag{7}\]  
where \(\mathbf{x}=vec(\underline{\mathbf{X}})\), \(\mathbf{r}=vec(\underline{\mathbf{R}})\), \(g_{i_{1}i_{2}i_{3}}\) are coefficients and \(\boldsymbol{\phi}_{i_{1}i_{2}i_{3}}=\mathbf{a}_{3}^{i_{3}}\otimes\mathbf{a}_{ 2}^{i_{2}}\otimes\mathbf{a}_{3}^{i_{3}}\) with \(\mathbf{a}_{n}^{i}\) denoting the \(i-\)column of matrix \(\mathbf{A}_{n}\).Tucker decomposition can provide data compression under two very different model assumptions leading to the following cases:  
**Low-rank Tucker decomposition:** when the core tensor is much smaller than the original, i.e., \(R_{n}\ll I_{n}\)[47; 48] (see Figure 1e)  
**Sparse Tucker decomposition:** when core tensor is of the same size or larger than tensor \(\underline{\mathbf{X}}\) but it is sparse as illustrated in Figure 1g. In this case, by looking at Equation (7), we conclude that the Sparse Tucker model corresponds to the classical Sparse Coding model of (4) with a dictionary that is obtained as the Kronecker product of three mode dictionaries, i.e., \(\boldsymbol{\Phi}=\mathbf{A}_{3}\otimes\mathbf{A}_{2}\otimes\mathbf{A}_{1}\)[49; 50]. Mode dictionaries can be chosen from classical sparsifying transforms such as wavelets, cosine transform and others or, if enough data is available, they can be learned from a dataset, which usually provides higher levels of sparsity and compression. A Kronecker dictionary learning algorithm was introduce in [50] and later a variant with orthogonality constraints was proposed in [51].