Fast and accurate randomized algorithms for low-rank tensor decompositions

Low-rank Tucker and CP tensor decompositions are powerful tools in data analytics . The widely used alternating least squares ( ALS ) method , which solves a sequence of over-determined least squares subproblems , is costly for large and sparse tensors . We propose a fast and accurate sketched ALS algorithm for Tucker decomposition , which solves a sequence of sketched rank-constrained linear least squares subproblems . Theoretical sketch size upper bounds are provided to achieve O ( ) relative error for each subproblem with two sketching techniques , TensorSketch and leverage score sampling . Experimental results show that this new ALS algorithm , combined with a new initialization scheme based on the randomized range finder , yields decomposition accuracy comparable to the standard higher-order orthogonal iteration ( HOOI ) algorithm . The new algorithm achieves up to 22.0 % relative decomposition residual improvement compared to the state-of-the-art sketched randomized algorithm for Tucker decomposition of various synthetic and real datasets . This Tucker-ALS algorithm is further used to accelerate CP decomposition , by using randomized Tucker compression followed by CP decomposition of the Tucker core tensor . Experimental results show that this algorithm not only converges faster , but also yields more accurate CP decompositions . 1 Introduction . Tensor decompositions [ 31 ] are general tools for compressing , approximating , as well as extracting important features from high dimensional data , and are widely used in both scientific computing [ 54 , 25 , 26 ] and machine learning [ 4 , 59 , 55 ] . In this paper , we focus on Tucker decomposition [ 67 ] and CANDECOMP/PARAFAC ( CP ) decomposition [ 24 , 23 ] . The alternating least squares ( ALS ) method is widely used to compute both decompositions . The ALS algorithm consists of sweeps , and each sweep updates every factor matrix once in a fixed order . The ALS method for Tucker decomposition , called the higher-order orthogonal iteration ( HOOI ) [ 5 , 16 , 31 ] , updates one of the factor matrices along with the core tensor at a time . Similarly , each update procedure in the ALS algorithm for CP decomposition ( CP-ALS ) updates one of the factor matrices . For both decompositions , solutions to each optimization subproblem guarantee decrease of the decomposition residual . In this work , we consider decomposition of order N tensors ( T ) that are large in dimension size ( s ) and can be potentially sparse . We focus on the problem of computing low-rank ( with target rank R s and R nnz ( T ) ) decompositions for such tensors via ALS , which is often used for extracting principal component information from large-scale datasets . For Tucker decomposition , ALS is bottlenecked by the operation called the tensor times matrix-chain ( TTMc ) . For CP decomposition , ALS is bottlenecked by the operation called the matricized tensor-times Khatri-Rao product ( MTTKRP ) . Both TTMc and MTTKRP have a per-sweep cost of Ω ( nnz ( T ) R ) [ 62 ] . Consequently , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the per-sweep costs of both HOOI and CP-ALS are proportional to the number of nonzeros in the tensor , which are expensive for large tensors with billions of nonzeros . Recent works have applied different randomized techniques to accelerate both CP [ 32 , 2 , 14 , 70 ] and Tucker decompositions [ 13 , 3 , 12 , 40 , 70 , 65 ] . For Tucker decomposition , these randomized algorithms apply sketching techniques to the higher-order singular value decomposition ( HOSVD ) . To do so , they calculate each factor matrix by applying randomized SVD on each matricization of the input tensor . These methods calculate the core tensor via TTMc among the input tensor and all the factor matrices , which incurs a cost of O ( nnz ( T ) R + sN−1R2 ) for sparse tensors and is still expensive . In addition , HOSVD generates decompositions that are generally less accurate compared to HOOI . Becker and Malik [ 39 ] introduce a sketched ALS algorithm for Tucker decomposition , which avoids the expensive cost of TTMc . Unlike the traditional HOOI , each sweep of this ALS scheme contains N + 1 subproblems , where only one of the factor matrices or the core tensor is updated in each subproblem . This scheme is easier to analyze theoretically , since each subproblem is an unconstrained linear least squares problem , which can be efficiently solved via sketching . However , the scheme produces decompositions that are generally less accurate than HOOI . 1.1 Our Contributions . In this work , we propose a new sketched ALS algorithm for Tucker decomposition . Different from Becker and Malik [ 39 ] , our ALS scheme is the same as HOOI , where one of the factor matrices along with the core tensor are updated in each subproblem . This guarantees the algorithm can reach the same accuracy as HOOI with sufficiently large sketch size . Experimental results show that it provides more accurate results compared to those in [ 39 ] . In this scheme , each subproblem is a sketched rank-constrained linear least squares problem , with the left-hand-side matrix with size sN−1 ×RN−1 composed of orthonormal columns . To the best of our knowledge , the relative error analysis of sketching techniques for this problem have not been discussed in the literature . Existing works either only provide sketch size upper bounds for the relaxed problem [ 57 ] , where rank constraint is relaxed with a nuclear norm constraint , or provide upper bounds for general constrained problems [ 69 ] . We provide tighter sketch size upper bounds to achieve O ( ) relative error with two state-of-the-art sketching techniques , TensorSketch [ 52 ] and leverage score sampling [ 18 ] . With leverage score sampling , our analysis shows that with probability at least 1 − δ , the sketch size of O ( RN−1/ ( 2δ ) ) is sufficient for results with O ( ) -relative error . With TensorSketch , the sketch size upper bound is O ( ( RN−1 · 3N−1 ) · ( RN−1 + 1/ 2 ) /δ ) , at least O ( 3N−1 ) times that for leverage score sampling . For both techniques , our bounds are at most O ( 1/ ) times the sketch size upper bounds for the unconstrained linear least squares problem . The upper bounds suggest that under the same accuracy criteria , leverage score sampling potentially needs smaller sketch size for each linear least squares problem and thus can be more efficient than TensorSketch . Therefore , with the same sketch size , the accuracy with leverage score sampling can be better . However , with the standard random initializations for factor matrices , leverage score sampling can perform poorly on tensors with high coherence [ 10 ] ( the orthogonal basis for the row space of each matricization of the input tensor has large row norm variability ) , making it less robust than TensorSketch . To improve the robustness of leverage score sampling , we introduce an algorithm that uses the randomized range finder ( RRF ) [ 22 ] to initialize the factor matrices . The initialization scheme uses the composition of CountSketch and Gaussian random matrix as the RRF embedding matrix , which only requires one pass over the non-zero elements of the input tensor . Our experimental results show that the leverage score sampling based randomized algorithm combined with this RRF scheme performs well on tensors with high coherence . For R s , our new sketching based algorithm for Tucker decomposition can also be used to accelerate CP decomposition . Tucker compression is performed first , and then CP decomposition is applied to the core tensor [ 70 , 9 , 20 ] . Since the per-sweep costs for both sketched Tucker-ALS and sketched CP-ALS are comparable , and Tucker-ALS often needs much fewer sweeps than CP-ALS ( Tucker-ALS typically converges in less than 5 sweeps based on our experiments ) , this Tucker + CP method can be more efficient than directly applying randomized CP decomposition [ 32 , 14 ] on the input tensor . In summary , this paper makes the following contributions . • We introduce a new sketched ALS algorithm for Tucker decomposition , which contains a sequence of sketched rank-constrained linear least squares subproblems . Experimental results show that the algorithm yields decomposition accuracy comparable to HOOI , and provides up to 22.0 % relative decomposition residual improvement compared to the previous randomized Tucker algorithm . • We provide theoretical upper bounds for the sketch size of both leverage score sampling and TensorSketch , which ensure that each sketched rank-constrained linear least squares incurs O ( ) relative error with high probability . • We provide detailed comparison of TensorSketch and leverage score sampling in terms of efficiency and accuracy . Our theoretical analysis shows that leverage score sampling is better in terms of both metrics . • We propose an initialization scheme based on RRF that improves the accuracy of leverage score sampling based sketching algorithm on tensors with high coherence . • We show that CP decomposition can be more efficiently and accurately calculated based on the sketched Tucker + CP method , compared to directly performing sketched CP-ALS on the input tensor . 2 Background . We introduce the notation used throughout the paper , and briefly review ALS algorithms for Tucker and CP decompositions , and TensorSketch as well as leverage score sampling in this section . We present additional backgrounds , including the pseudo-codes of ALS algorithms for both Tucker and CP decompositions , and the previous work in Appendix A . 2.1 Notation . Our analysis makes use of tensor algebra in both element-wise equations and specialized notation for tensor operations [ 31 ] . Vectors are denoted with bold lowercase Roman letters ( e.g. , v ) , matrices are denoted with bold uppercase Roman letters ( e.g. , M ) , and tensors are denoted with bold calligraphic font ( e.g. , T ) . An order N tensor corresponds to an N -dimensional array . Elements of vectors , matrices , and tensors are denoted in parentheses , e.g. , v ( i ) for a vector v , M ( i , j ) for a matrix M , and T ( i , j , k , l ) for an order 4 tensor T . The ith column of M is denoted by M ( : , i ) , and the ith row is denoted by M ( i , : ) . Parenthesized superscripts are used to label different vectors , matrices and tensors ( e.g . T ( 1 ) and T ( 2 ) are unrelated tensors ) . Number of nonzeros of the tensor T is denoted by nnz ( T ) . The pseudo-inverse of matrix A is denoted with A† . The Hadamard product of two matrices is denoted with ∗ . The outer product of two or more vectors is denoted with ◦ . The Kronecker product of two vectors/matrices is denoted with⊗ . For matrices A ∈ Rm×k and B ∈ Rn×k , their Khatri-Rao product results in a matrix of size ( mn ) × k defined by A B = [ A ( : , 1 ) ⊗ B ( : , 1 ) , . . . , A ( : , k ) ⊗B ( : , k ) ] . The mode-n tensor times matrix of an order N tensor T ∈ Rs1×···×sN with a matrix A ∈ RJ×sn is denoted by T ×n A , whose output size is s1 × · · · × sn−1 × J × sn+1 × · · · × sN . Matricization is the process of unfolding a tensor into a matrix . The mode-n matricized version of T is denoted by T ( n ) ∈ Rsn×K where K = ∏N m=1 , m6=n sm . We use [ N ] to denote { 1 , . . . , N } . Õ denotes the asymptotic cost with logarithmic factors ignored . Tucker decomposition with ALS . Throughout the analysis we assume the input tensor has order N and size s× · · · × s , and the Tucker ranks are R× · · · ×R . Tucker decomposition approximates a tensor by a core tensor contracted along each mode with matrices that have orthonormal columns . The goal of Tucker decomposition is to minimize the objective function , f ( C , A ( 1 ) , . . . , A ( N ) ) : = 1 2 ∥∥T −C ×1 A ( 1 ) ×2 A ( 2 ) · · · ×N A ( N ) ∥∥2F . The core tensor C is of order N with dimensions R× · · · × R. Each matrix A ( n ) ∈ Rs×R for n ∈ [ N ] has orthonormal columns . The ALS method for Tucker decomposition [ 5 , 16 , 31 ] , called the higher-order orthogonal iteration ( HOOI ) , proceeds by updating one of the factor matrices along with the core tensor at a time . The nth subproblem can be written as min C , A ( n ) 1 2 ∥∥∥P ( n ) CT ( n ) A ( n ) T −TT ( n ) ∥∥∥2 F , ( 2.1 ) where P ( n ) = A ( 1 ) ⊗ · · · ⊗A ( n−1 ) ⊗A ( n+1 ) ⊗ · · · ⊗A ( N ) . This problem can be formulated as a rank-constrained linear least squares problem , min B ( n ) 1 2 ∥∥∥P ( n ) B ( n ) −TT ( n ) ∥∥∥2 F , such that rank ( B ( n ) ) ≤ R. ( 2.2 ) A ( n ) corresponds to the right singular vectors of the optimal B ( n ) , while CT ( n ) = B ( n ) A ( n ) . Since P ( n ) contains orthonormal columns , the optimal B ( n ) can be obtained by calculating the Tensor Times Matrix-chain ( TTMc ) , Y ( n ) = T ×1 A ( 1 ) T · · · ×n−1 A ( n−1 ) T ×n+1 A ( n+1 ) T · · · ×N A ( N ) T , ( 2.3 ) and taking B ( n ) to be the transpose of the mode-n matricized Y ( n ) , Y ( n ) T ( n ) . Calculating Y ( n ) costs O ( sNR ) for dense tensors andO ( nnz ( T ) RN−1 ) for sparse tensors . Before the HOOI procedure , the factor matrices are often initialized with the higher-order singular value decomposition ( HOSVD ) [ 15 , 67 ] . HOSVD computes the truncated SVD of each T ( n ) ≈ U ( n ) Σ ( n ) V ( n ) T , and sets A ( n ) = U ( n ) for n ∈ [ N ] . If performing SVD via randomized SVD [ 22 ] , updating A ( n ) for each mode costs O ( sNR ) for dense tensors , and costs O ( sN−1R2 + nnz ( T ) R ) for sparse tensors . CP decomposition with ALS . CP tensor decomposition [ 24 , 23 ] decomposes the input tensor into a sum of outer products of vectors . Throughout analysis we assume the input tensor has order N and size s × · · · × s , and the CP rank is R. The goal of CP decomposition is to minimize the objective function , f ( A ( 1 ) , . . . , A ( N ) ) : = 12 ∥∥∥T −∑Rr=1 A ( 1 ) ( : , r ) ◦ · · · ◦A ( N ) ( : , r ) ∥∥∥2 F , where A ( i ) ∈ Rs×R for i ∈ [ N ] are called factor matrices . CP-ALS is the mostly widely used algorithm to get the factor matrices . In each ALS sweep , we solve N subproblems , and the objective for the update of A ( n ) , with n ∈ [ N ] , is expressed as , A ( n ) = arg min A 1 2 ∥∥∥P ( n ) AT −XT ( n ) ∥∥∥2 F , ( 2.4 ) where P ( n ) = A ( 1 ) · · · A ( n−1 ) A ( n+1 ) · · · A ( N ) . Solving the linear least squares problem above has a cost of O ( sNR ) . For instance , when solving via normal equations the term P ( n ) TXT ( n ) , which is called MTTKRP , needs to be calculated , and it costsO ( sNR ) for dense tensors andO ( nnz ( T ) R ) for sparse tensors . A major disadvantage of CP-ALS is its slow convergence . There are many cases where CP-ALS takes a large number of sweeps to converge when high resolution is necessary [ 42 ] . When R < s , the procedure can be accelerated by performing Tucker-ALS first , which typically converges in fewer sweeps , and then computing a CP decomposition of the core tensor [ 11 , 70 , 9 ] , which only has O ( RN ) elements . TensorSketch and leverage score sampling . In this paper , we sketch the linear least squares problems using both TensorSketch and leverage score sampling . TensorSketch is a special type of CountSketch , where the hash map is restricted to a specific format to allow fast multiplication of the sketching matrix with the chain of Kronecker products . Leverage score sampling picks important rows based on leverage scores to form the sampled/sketched problem . Both algorithms can be efficiently applied to a chain of Kronecker products , and the detailed analysis is presented in Appendix B . In the paper , we test two forms of leverage score sampling , random sampling , where we perform importance random sampling based on leverage scores , and deterministic sampling [ 28 ] , where we deterministically sample rows having the largest leverage scores . Both ideas are also used in [ 32 ] for randomized CP decomposition . Papailiopoulos et al . [ 53 ] show that if the leverage scores follow a moderately steep power-law decay , then deterministic sampling can be provably as efficient and even better than the random sampling . We compare both leverage score sampling techniques in Section 5 .

Summary: This is paper presents a new randomized sketching algorithm for computing a low rank Tucker and CP decomposition of tensors. The prospered method is based on using sketching to solve the rank-constrained linear least squares subproblems that appear in the popular alternating least squares (ALS) method. Theoretical analysis is presented that give the sampling complexity needed to solve each rank-constrained subproblems to a certain relative error guarantee. Results are presented for two types of sketching, TensorSketch and leverage score sampling. The rank-constrained linear least squares is solved using the randomized SVD. Many numerical results are presented that illustrate the performance of the proposed method.