Fast Tucker Rank Reduction for Non-Negative Tensors Using Mean-Field Approximation

1 Introduction . A multidimensional array , or tensor , is a fundamental data structure in machine learning and statistical data analysis , and extraction of the essential information contained in tensors has been studied extensively [ 14 , 20 ] . For second-order tensors – that is , matrices – low-rank approximation by singular value decomposition ( SVD ) is well established [ 13 ] . SVD always provides the best low-rank approximation in the sense of arbitrary unitarily invariant norms [ 30 ] . In contrast , the problem of low-rank approximation becomes much more challenging for tensors higher than the second order , where the question of how to define the rank of tensors is even nontrivial . To date , various types of ranks – the CP-rank [ 17 , 24 ] , the Tucker rank [ 9 , 39 ] , and the tubal rank [ 29 ] – have been proposed , and low-rank approximation of tensors in terms of one of the above two ranks has been widely studied . Furthermore , non-negative low-rank approximation has also been developed , not only for matrices such as NMF [ 26 ] , but also for tensors [ 27 ] . In particular , non-negative Tucker decomposition ( NTD ) [ 21 ] and its efficient variant lraSNTD [ 43 ] approximate a given non-negative tensor by a tensor with the lower Tucker rank . While these approximations have been widely used in various domains such as image classification [ 23 ] , recommendation [ 37 ] , and denoising [ 12 ] , efficient low-rank approximation remains fundamentally challenging . Even the simplest case , the rank-1 approximation in terms of minimizing the Least Squares ( LS ) error between a given tensor and a low-rank tensor , is known to be NP-hard [ 16 ] . Various methods have been developed to efficiently find approximate solutions in polynomial runtime [ 7 , 8 , 10 , 25 , 42 ] . If we use the Kullback–Leibler ( KL ) divergence instead of the LS error as a cost function , we can alleviate the problem as the best rank-1 approximation can be obtained in the closed formula [ 19 ] . However , the general case of low-rank approximation in terms of the KL divergence is also still under development . In this paper , we present a fast low-Tucker-rank approximation method for non-negative tensors . To date , the majority of low-rank approximation methods are based on gradient decent using the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . derivative of the cost function , which often requires careful tuning of initialization and/or a tolerance threshold . In contrast , our method is not based on a gradient method ; the solution is directly obtained based on a closed formula , which we derive from information geometric treatment of tensors . Through an alternative parameterization of tensors by treating them as probability distributions in a statistical manifold , we theoretically provide a sufficient condition for such parameters , called the bingo rule , to reduce Tucker ranks of tensors . We then show that low-rank approximation is achieved by m-projection ; this is one of the two canonical projections in information geometry [ 2 ] , where a distribution ( corresponding to a given non-negative tensor ) is projected onto the subspace restricted by the bingo rule ( corresponding to the set of non-negative low-rank tensors ) . The key insight is that rank-1 approximation for non-negative tensors can be exactly solved by a mean-field approximation , a well-established method in physics that approximates a joint distribution by independent distributions [ 40 ] , as we can represent any non-negative rank-1 tensor by a product of independent distributions . Moreover , we show that the bingo rule , our sufficient condition for tensor Tucker rank reduction , can be achieved by iterative applications of the mean-field approximation . This , combined with the fact that mean-field approximation is computed bym-projection in the closed form , enables us to derive our fast low-Tucker-rank approximation method without using a gradient method . Our theoretical analysis has a close relationship to [ 36 ] , whose proposal , called Legendre decomposition , also uses information geometric parameterization of tensors and solves the problem of tensor decomposition by a projection onto a subspace . Although we use the same information geometric formulation of tensors , they did not provide any connection to the Tucker ranks , and Tucker rank reduction is not guaranteed by their approach . A limitation of our method is that it only treats non-negative tensors and can not handle tensors with zero or negative values . Although experimental results show the usefulness of our method , even for tensors including zeros , we always assume that every element of an input tensor is strictly positive in our theoretical discussion . 2 The Proposed Rank Reduction Algorithm . We propose a Tucker rank reduction algorithm , called the Legendre Tucker rank reduction ( LTR ) , which transforms a given non-negative tensor P ∈ RI1×···×Id≥0 into a tensor that approximates P with arbitrary reduced Tucker rank ( r1 , . . . , rd ) specified by the user . The term “ Legendre ” of our algorithm comes from the fact that the theoretical support of our algorithm uses parameterization of tensors based on the Legendre transformation , which will be clarified in Section 3 . 2.1 Problem setup and notation . First we define the Tucker rank of tensors and formulate the problem of non-negative Tucker rank reduction . The Tucker rank of a dth-order tensor P ∈ RI1×···×Id is defined as a tuple ( Rank ( P ( 1 ) ) , . . . , Rank ( P ( d ) ) ) , where each P ( k ) ∈ RIk× ∏ m6=k Im is the mode-k expansion of the tensor P and Rank ( P ( k ) ) denotes the matrix rank of P ( k ) [ 11 , 15 , 39 ] . See Supplement for definition of the mode-k expansion . If the Tucker rank of a tensor P is ( r1 , . . . , rd ) , it can be always decomposed as P = ∑r1 i1=1 · · · ∑rd id=1 Gi1 , ... , ida ( 1 ) i1 ⊗ a ( 2 ) i2 ⊗ · · · ⊗ a ( d ) id with a tensor G ∈ Rr1×···×rd , called the core tensor of P , and vectors a ( k ) ik ∈ RIk , ik ∈ { 1 , . . . , rk } , for each k ∈ { 1 , . . . , d } , where ⊗ denotes the Kronecker product [ 24 ] . If every element of the Tucker rank is the same as each other element , it coincides with the CP rank . In this paper , we say that a tensor is rank-1 if its Tucker rank is ( 1 , . . . , 1 ) . The problem with non-negative Tucker rank reduction is approximating a given non-negative tensor by a non-negative lower Tucker rank tensor . We denote by [ d ] = { 1 , 2 , . . . , d } for a positive integer d and denote by Pa ( k ) : b ( k ) the subtensor obtained by fixing the range of kth index to only from a to b . 2.2 The LTR Algorithm . In LTR , we use the rank-1 approximation method that always finds the rank-1 tensor that minimizes the KL divergence from an input tensor [ 19 ] . The optimal rank-1 tensor P of P is given by P = λs ( 1 ) ⊗ s ( 2 ) ⊗ · · · ⊗ s ( d ) , ( 1 ) Algorithm 1 : input : Tensor P , target Tucker rank r = ( r1 , . . . , rd ) output : Rank reduced tensor Q LTR ( P , r ) ( I1 , . . . , Id ) ← the size of the input tensor P foreach k = 1 , . . . , d do Construct { c1 , . . . , crk } ⊆ [ Ik ] by random sampling from [ Ik ] without replacement , where we always assume that c1 = 1 and ci < ci+1 . foreach l = 1 , . . . , rk do if cl 6= cl+1 − 1 then Replace the subtensor P c ( k ) l : c ( k ) l+1−1 of P by its rank-1 approximation as P c ( k ) l : c ( k ) l+1−1 ← BESTRANK1 ( P c ( k ) l : c ( k ) l+1−1 ) Q ← P return Q BESTRANK1 ( P ) ( I1 , . . . , Id ) ← the size of the input tensor P foreach k = 1 , . . . , d do foreach ik = 1 , . . . , Ik do s ( k ) ik ←∑I1i1=1 · · ·∑Ik−1ik−1=1∑Ik+1ik+1=1 · · ·∑Idid=1 Pi1 , ... ik−1 , ik , ik+1 , ... , id λ← sum of all elements of P P ← λs ( 1 ) ⊗ s ( 2 ) ⊗ · · · ⊗ s ( d ) return P where each s ( k ) = ( s ( k ) 1 , . . . , s ( k ) Ik ) with k ∈ [ d ] is defined as s ( k ) ik = I1∑ i1=1 · · · Ik−1∑ ik−1=1 Ik+1∑ ik+1=1 · · · Id∑ id=1 Pi1 , ... , id and λ is the ( 1− d ) -th power of the sum of all elements of P . Now we introduce LTR , which iteratively applies the above rank-1 approximation to subtensors of a tensor P ∈ RI1×···×Id . When we reduce the Tucker rank of P to ( r1 , . . . , rd ) , LTR performs the following two steps for each k ∈ [ d ] : Step 1 : We construct C = { c1 , . . . , crk } ⊆ [ Ik ] by random sampling from [ Ik ] without replacement , where we always assume that c1 = 1 and cl < cl+1 for every l ∈ [ rk − 1 ] . Step 2 : For each l ∈ [ rk ] , if cl 6= cl+1 − 1 holds , we replace the subtensor Pc ( k ) l : c ( k ) l+1−1 of P by its rank-1 approximation obtained by Equation ( 1 ) . Step 1 of LTR requires O ( r1 + r2 + · · ·+ rd ) since we only need to sample integers from 1 , 2 , . . . , Ik for each k ∈ [ d ] using the Fisher-Yates method . Since the above procedure repeats the best rank-1 approximation at most r1r2 . . . rd times , the worst computational complexity of LTR is O ( r1r2 . . . rdI1I2 . . . Id ) . The choice of C in Step 1 is arbitrary , which means that another strategy can be used . For example , if we know that some parts of an input tensor are less important than others , we can directly choose these indices for C instead of random sampling to obtain a more accurate reconstructed tensor . We provide the algorithm of LTR in algorithmic format in Algorithm 1 .

Summary: This paper derives a tensor decomposition method using mean-field approximation and information geometry. The main claim of this paper is the speed of computation compared to some basic baseline methods. An interesting direction but there are several directions that the paper can improve further.