Fast and accurate randomized algorithms for low-rank tensor decompositions

Low-rank Tucker and CP tensor decompositions are powerful tools in data analytics . The widely used alternating least squares ( ALS ) method , which solves a sequence of over-determined least squares subproblems , is costly for large and sparse tensors . We propose a fast and accurate sketched ALS algorithm for Tucker decomposition , which solves a sequence of sketched rank-constrained linear least squares subproblems . Theoretical sketch size upper bounds are provided to achieve O ( ) relative error for each subproblem with two sketching techniques , TensorSketch and leverage score sampling . Experimental results show that this new ALS algorithm , combined with a new initialization scheme based on the randomized range finder , yields decomposition accuracy comparable to the standard higher-order orthogonal iteration ( HOOI ) algorithm . The new algorithm achieves up to 22.0 % relative decomposition residual improvement compared to the state-of-the-art sketched randomized algorithm for Tucker decomposition of various synthetic and real datasets . This Tucker-ALS algorithm is further used to accelerate CP decomposition , by using randomized Tucker compression followed by CP decomposition of the Tucker core tensor . Experimental results show that this algorithm not only converges faster , but also yields more accurate CP decompositions . 1 Introduction . Tensor decompositions [ 31 ] are general tools for compressing , approximating , as well as extracting important features from high dimensional data , and are widely used in both scientific computing [ 54 , 25 , 26 ] and machine learning [ 4 , 59 , 55 ] . In this paper , we focus on Tucker decomposition [ 67 ] and CANDECOMP/PARAFAC ( CP ) decomposition [ 24 , 23 ] . The alternating least squares ( ALS ) method is widely used to compute both decompositions . The ALS algorithm consists of sweeps , and each sweep updates every factor matrix once in a fixed order . The ALS method for Tucker decomposition , called the higher-order orthogonal iteration ( HOOI ) [ 5 , 16 , 31 ] , updates one of the factor matrices along with the core tensor at a time . Similarly , each update procedure in the ALS algorithm for CP decomposition ( CP-ALS ) updates one of the factor matrices . For both decompositions , solutions to each optimization subproblem guarantee decrease of the decomposition residual . In this work , we consider decomposition of order N tensors ( T ) that are large in dimension size ( s ) and can be potentially sparse . We focus on the problem of computing low-rank ( with target rank R s and R nnz ( T ) ) decompositions for such tensors via ALS , which is often used for extracting principal component information from large-scale datasets . For Tucker decomposition , ALS is bottlenecked by the operation called the tensor times matrix-chain ( TTMc ) . For CP decomposition , ALS is bottlenecked by the operation called the matricized tensor-times Khatri-Rao product ( MTTKRP ) . Both TTMc and MTTKRP have a per-sweep cost of Ω ( nnz ( T ) R ) [ 62 ] . Consequently , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the per-sweep costs of both HOOI and CP-ALS are proportional to the number of nonzeros in the tensor , which are expensive for large tensors with billions of nonzeros . Recent works have applied different randomized techniques to accelerate both CP [ 32 , 2 , 14 , 70 ] and Tucker decompositions [ 13 , 3 , 12 , 40 , 70 , 65 ] . For Tucker decomposition , these randomized algorithms apply sketching techniques to the higher-order singular value decomposition ( HOSVD ) . To do so , they calculate each factor matrix by applying randomized SVD on each matricization of the input tensor . These methods calculate the core tensor via TTMc among the input tensor and all the factor matrices , which incurs a cost of O ( nnz ( T ) R + sN−1R2 ) for sparse tensors and is still expensive . In addition , HOSVD generates decompositions that are generally less accurate compared to HOOI . Becker and Malik [ 39 ] introduce a sketched ALS algorithm for Tucker decomposition , which avoids the expensive cost of TTMc . Unlike the traditional HOOI , each sweep of this ALS scheme contains N + 1 subproblems , where only one of the factor matrices or the core tensor is updated in each subproblem . This scheme is easier to analyze theoretically , since each subproblem is an unconstrained linear least squares problem , which can be efficiently solved via sketching . However , the scheme produces decompositions that are generally less accurate than HOOI . 1.1 Our Contributions . In this work , we propose a new sketched ALS algorithm for Tucker decomposition . Different from Becker and Malik [ 39 ] , our ALS scheme is the same as HOOI , where one of the factor matrices along with the core tensor are updated in each subproblem . This guarantees the algorithm can reach the same accuracy as HOOI with sufficiently large sketch size . Experimental results show that it provides more accurate results compared to those in [ 39 ] . In this scheme , each subproblem is a sketched rank-constrained linear least squares problem , with the left-hand-side matrix with size sN−1 ×RN−1 composed of orthonormal columns . To the best of our knowledge , the relative error analysis of sketching techniques for this problem have not been discussed in the literature . Existing works either only provide sketch size upper bounds for the relaxed problem [ 57 ] , where rank constraint is relaxed with a nuclear norm constraint , or provide upper bounds for general constrained problems [ 69 ] . We provide tighter sketch size upper bounds to achieve O ( ) relative error with two state-of-the-art sketching techniques , TensorSketch [ 52 ] and leverage score sampling [ 18 ] . With leverage score sampling , our analysis shows that with probability at least 1 − δ , the sketch size of O ( RN−1/ ( 2δ ) ) is sufficient for results with O ( ) -relative error . With TensorSketch , the sketch size upper bound is O ( ( RN−1 · 3N−1 ) · ( RN−1 + 1/ 2 ) /δ ) , at least O ( 3N−1 ) times that for leverage score sampling . For both techniques , our bounds are at most O ( 1/ ) times the sketch size upper bounds for the unconstrained linear least squares problem . The upper bounds suggest that under the same accuracy criteria , leverage score sampling potentially needs smaller sketch size for each linear least squares problem and thus can be more efficient than TensorSketch . Therefore , with the same sketch size , the accuracy with leverage score sampling can be better . However , with the standard random initializations for factor matrices , leverage score sampling can perform poorly on tensors with high coherence [ 10 ] ( the orthogonal basis for the row space of each matricization of the input tensor has large row norm variability ) , making it less robust than TensorSketch . To improve the robustness of leverage score sampling , we introduce an algorithm that uses the randomized range finder ( RRF ) [ 22 ] to initialize the factor matrices . The initialization scheme uses the composition of CountSketch and Gaussian random matrix as the RRF embedding matrix , which only requires one pass over the non-zero elements of the input tensor . Our experimental results show that the leverage score sampling based randomized algorithm combined with this RRF scheme performs well on tensors with high coherence . For R s , our new sketching based algorithm for Tucker decomposition can also be used to accelerate CP decomposition . Tucker compression is performed first , and then CP decomposition is applied to the core tensor [ 70 , 9 , 20 ] . Since the per-sweep costs for both sketched Tucker-ALS and sketched CP-ALS are comparable , and Tucker-ALS often needs much fewer sweeps than CP-ALS ( Tucker-ALS typically converges in less than 5 sweeps based on our experiments ) , this Tucker + CP method can be more efficient than directly applying randomized CP decomposition [ 32 , 14 ] on the input tensor . In summary , this paper makes the following contributions . • We introduce a new sketched ALS algorithm for Tucker decomposition , which contains a sequence of sketched rank-constrained linear least squares subproblems . Experimental results show that the algorithm yields decomposition accuracy comparable to HOOI , and provides up to 22.0 % relative decomposition residual improvement compared to the previous randomized Tucker algorithm . • We provide theoretical upper bounds for the sketch size of both leverage score sampling and TensorSketch , which ensure that each sketched rank-constrained linear least squares incurs O ( ) relative error with high probability . • We provide detailed comparison of TensorSketch and leverage score sampling in terms of efficiency and accuracy . Our theoretical analysis shows that leverage score sampling is better in terms of both metrics . • We propose an initialization scheme based on RRF that improves the accuracy of leverage score sampling based sketching algorithm on tensors with high coherence . • We show that CP decomposition can be more efficiently and accurately calculated based on the sketched Tucker + CP method , compared to directly performing sketched CP-ALS on the input tensor . 2 Background . We introduce the notation used throughout the paper , and briefly review ALS algorithms for Tucker and CP decompositions , and TensorSketch as well as leverage score sampling in this section . We present additional backgrounds , including the pseudo-codes of ALS algorithms for both Tucker and CP decompositions , and the previous work in Appendix A . 2.1 Notation . Our analysis makes use of tensor algebra in both element-wise equations and specialized notation for tensor operations [ 31 ] . Vectors are denoted with bold lowercase Roman letters ( e.g. , v ) , matrices are denoted with bold uppercase Roman letters ( e.g. , M ) , and tensors are denoted with bold calligraphic font ( e.g. , T ) . An order N tensor corresponds to an N -dimensional array . Elements of vectors , matrices , and tensors are denoted in parentheses , e.g. , v ( i ) for a vector v , M ( i , j ) for a matrix M , and T ( i , j , k , l ) for an order 4 tensor T . The ith column of M is denoted by M ( : , i ) , and the ith row is denoted by M ( i , : ) . Parenthesized superscripts are used to label different vectors , matrices and tensors ( e.g . T ( 1 ) and T ( 2 ) are unrelated tensors ) . Number of nonzeros of the tensor T is denoted by nnz ( T ) . The pseudo-inverse of matrix A is denoted with A† . The Hadamard product of two matrices is denoted with ∗ . The outer product of two or more vectors is denoted with ◦ . The Kronecker product of two vectors/matrices is denoted with⊗ . For matrices A ∈ Rm×k and B ∈ Rn×k , their Khatri-Rao product results in a matrix of size ( mn ) × k defined by A B = [ A ( : , 1 ) ⊗ B ( : , 1 ) , . . . , A ( : , k ) ⊗B ( : , k ) ] . The mode-n tensor times matrix of an order N tensor T ∈ Rs1×···×sN with a matrix A ∈ RJ×sn is denoted by T ×n A , whose output size is s1 × · · · × sn−1 × J × sn+1 × · · · × sN . Matricization is the process of unfolding a tensor into a matrix . The mode-n matricized version of T is denoted by T ( n ) ∈ Rsn×K where K = ∏N m=1 , m6=n sm . We use [ N ] to denote { 1 , . . . , N } . Õ denotes the asymptotic cost with logarithmic factors ignored . Tucker decomposition with ALS . Throughout the analysis we assume the input tensor has order N and size s× · · · × s , and the Tucker ranks are R× · · · ×R . Tucker decomposition approximates a tensor by a core tensor contracted along each mode with matrices that have orthonormal columns . The goal of Tucker decomposition is to minimize the objective function , f ( C , A ( 1 ) , . . . , A ( N ) ) : = 1 2 ∥∥T −C ×1 A ( 1 ) ×2 A ( 2 ) · · · ×N A ( N ) ∥∥2F . The core tensor C is of order N with dimensions R× · · · × R. Each matrix A ( n ) ∈ Rs×R for n ∈ [ N ] has orthonormal columns . The ALS method for Tucker decomposition [ 5 , 16 , 31 ] , called the higher-order orthogonal iteration ( HOOI ) , proceeds by updating one of the factor matrices along with the core tensor at a time . The nth subproblem can be written as min C , A ( n ) 1 2 ∥∥∥P ( n ) CT ( n ) A ( n ) T −TT ( n ) ∥∥∥2 F , ( 2.1 ) where P ( n ) = A ( 1 ) ⊗ · · · ⊗A ( n−1 ) ⊗A ( n+1 ) ⊗ · · · ⊗A ( N ) . This problem can be formulated as a rank-constrained linear least squares problem , min B ( n ) 1 2 ∥∥∥P ( n ) B ( n ) −TT ( n ) ∥∥∥2 F , such that rank ( B ( n ) ) ≤ R. ( 2.2 ) A ( n ) corresponds to the right singular vectors of the optimal B ( n ) , while CT ( n ) = B ( n ) A ( n ) . Since P ( n ) contains orthonormal columns , the optimal B ( n ) can be obtained by calculating the Tensor Times Matrix-chain ( TTMc ) , Y ( n ) = T ×1 A ( 1 ) T · · · ×n−1 A ( n−1 ) T ×n+1 A ( n+1 ) T · · · ×N A ( N ) T , ( 2.3 ) and taking B ( n ) to be the transpose of the mode-n matricized Y ( n ) , Y ( n ) T ( n ) . Calculating Y ( n ) costs O ( sNR ) for dense tensors andO ( nnz ( T ) RN−1 ) for sparse tensors . Before the HOOI procedure , the factor matrices are often initialized with the higher-order singular value decomposition ( HOSVD ) [ 15 , 67 ] . HOSVD computes the truncated SVD of each T ( n ) ≈ U ( n ) Σ ( n ) V ( n ) T , and sets A ( n ) = U ( n ) for n ∈ [ N ] . If performing SVD via randomized SVD [ 22 ] , updating A ( n ) for each mode costs O ( sNR ) for dense tensors , and costs O ( sN−1R2 + nnz ( T ) R ) for sparse tensors . CP decomposition with ALS . CP tensor decomposition [ 24 , 23 ] decomposes the input tensor into a sum of outer products of vectors . Throughout analysis we assume the input tensor has order N and size s × · · · × s , and the CP rank is R. The goal of CP decomposition is to minimize the objective function , f ( A ( 1 ) , . . . , A ( N ) ) : = 12 ∥∥∥T −∑Rr=1 A ( 1 ) ( : , r ) ◦ · · · ◦A ( N ) ( : , r ) ∥∥∥2 F , where A ( i ) ∈ Rs×R for i ∈ [ N ] are called factor matrices . CP-ALS is the mostly widely used algorithm to get the factor matrices . In each ALS sweep , we solve N subproblems , and the objective for the update of A ( n ) , with n ∈ [ N ] , is expressed as , A ( n ) = arg min A 1 2 ∥∥∥P ( n ) AT −XT ( n ) ∥∥∥2 F , ( 2.4 ) where P ( n ) = A ( 1 ) · · · A ( n−1 ) A ( n+1 ) · · · A ( N ) . Solving the linear least squares problem above has a cost of O ( sNR ) . For instance , when solving via normal equations the term P ( n ) TXT ( n ) , which is called MTTKRP , needs to be calculated , and it costsO ( sNR ) for dense tensors andO ( nnz ( T ) R ) for sparse tensors . A major disadvantage of CP-ALS is its slow convergence . There are many cases where CP-ALS takes a large number of sweeps to converge when high resolution is necessary [ 42 ] . When R < s , the procedure can be accelerated by performing Tucker-ALS first , which typically converges in fewer sweeps , and then computing a CP decomposition of the core tensor [ 11 , 70 , 9 ] , which only has O ( RN ) elements . TensorSketch and leverage score sampling . In this paper , we sketch the linear least squares problems using both TensorSketch and leverage score sampling . TensorSketch is a special type of CountSketch , where the hash map is restricted to a specific format to allow fast multiplication of the sketching matrix with the chain of Kronecker products . Leverage score sampling picks important rows based on leverage scores to form the sampled/sketched problem . Both algorithms can be efficiently applied to a chain of Kronecker products , and the detailed analysis is presented in Appendix B . In the paper , we test two forms of leverage score sampling , random sampling , where we perform importance random sampling based on leverage scores , and deterministic sampling [ 28 ] , where we deterministically sample rows having the largest leverage scores . Both ideas are also used in [ 32 ] for randomized CP decomposition . Papailiopoulos et al . [ 53 ] show that if the leverage scores follow a moderately steep power-law decay , then deterministic sampling can be provably as efficient and even better than the random sampling . We compare both leverage score sampling techniques in Section 5 .

Summary: The paper proposed a fast and accurate sketching based ALS algorithm for Tucker decomposition. The method is composed of a sequence of sketched rank-constrained linear least squares subproblems. In addition, the paper proved that the sketch sizes of TensorSketch and leverage score sampling that are sufficient for the relative residual norm error of the problems to be bounded by $O(\epsilon)$ with at least $1-\delta$ probability.  The experiments show that proposed algorithm is more accurate than the existing sketching based randomized algorithm for Tucker decomposition. The algorithm can also be extended to CP decomposition effectively.