Fast Tucker Rank Reduction for Non-Negative Tensors Using Mean-Field Approximation

1 Introduction . A multidimensional array , or tensor , is a fundamental data structure in machine learning and statistical data analysis , and extraction of the essential information contained in tensors has been studied extensively [ 14 , 20 ] . For second-order tensors – that is , matrices – low-rank approximation by singular value decomposition ( SVD ) is well established [ 13 ] . SVD always provides the best low-rank approximation in the sense of arbitrary unitarily invariant norms [ 30 ] . In contrast , the problem of low-rank approximation becomes much more challenging for tensors higher than the second order , where the question of how to define the rank of tensors is even nontrivial . To date , various types of ranks – the CP-rank [ 17 , 24 ] , the Tucker rank [ 9 , 39 ] , and the tubal rank [ 29 ] – have been proposed , and low-rank approximation of tensors in terms of one of the above two ranks has been widely studied . Furthermore , non-negative low-rank approximation has also been developed , not only for matrices such as NMF [ 26 ] , but also for tensors [ 27 ] . In particular , non-negative Tucker decomposition ( NTD ) [ 21 ] and its efficient variant lraSNTD [ 43 ] approximate a given non-negative tensor by a tensor with the lower Tucker rank . While these approximations have been widely used in various domains such as image classification [ 23 ] , recommendation [ 37 ] , and denoising [ 12 ] , efficient low-rank approximation remains fundamentally challenging . Even the simplest case , the rank-1 approximation in terms of minimizing the Least Squares ( LS ) error between a given tensor and a low-rank tensor , is known to be NP-hard [ 16 ] . Various methods have been developed to efficiently find approximate solutions in polynomial runtime [ 7 , 8 , 10 , 25 , 42 ] . If we use the Kullback–Leibler ( KL ) divergence instead of the LS error as a cost function , we can alleviate the problem as the best rank-1 approximation can be obtained in the closed formula [ 19 ] . However , the general case of low-rank approximation in terms of the KL divergence is also still under development . In this paper , we present a fast low-Tucker-rank approximation method for non-negative tensors . To date , the majority of low-rank approximation methods are based on gradient decent using the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . derivative of the cost function , which often requires careful tuning of initialization and/or a tolerance threshold . In contrast , our method is not based on a gradient method ; the solution is directly obtained based on a closed formula , which we derive from information geometric treatment of tensors . Through an alternative parameterization of tensors by treating them as probability distributions in a statistical manifold , we theoretically provide a sufficient condition for such parameters , called the bingo rule , to reduce Tucker ranks of tensors . We then show that low-rank approximation is achieved by m-projection ; this is one of the two canonical projections in information geometry [ 2 ] , where a distribution ( corresponding to a given non-negative tensor ) is projected onto the subspace restricted by the bingo rule ( corresponding to the set of non-negative low-rank tensors ) . The key insight is that rank-1 approximation for non-negative tensors can be exactly solved by a mean-field approximation , a well-established method in physics that approximates a joint distribution by independent distributions [ 40 ] , as we can represent any non-negative rank-1 tensor by a product of independent distributions . Moreover , we show that the bingo rule , our sufficient condition for tensor Tucker rank reduction , can be achieved by iterative applications of the mean-field approximation . This , combined with the fact that mean-field approximation is computed bym-projection in the closed form , enables us to derive our fast low-Tucker-rank approximation method without using a gradient method . Our theoretical analysis has a close relationship to [ 36 ] , whose proposal , called Legendre decomposition , also uses information geometric parameterization of tensors and solves the problem of tensor decomposition by a projection onto a subspace . Although we use the same information geometric formulation of tensors , they did not provide any connection to the Tucker ranks , and Tucker rank reduction is not guaranteed by their approach . A limitation of our method is that it only treats non-negative tensors and can not handle tensors with zero or negative values . Although experimental results show the usefulness of our method , even for tensors including zeros , we always assume that every element of an input tensor is strictly positive in our theoretical discussion . 2 The Proposed Rank Reduction Algorithm . We propose a Tucker rank reduction algorithm , called the Legendre Tucker rank reduction ( LTR ) , which transforms a given non-negative tensor P ∈ RI1×···×Id≥0 into a tensor that approximates P with arbitrary reduced Tucker rank ( r1 , . . . , rd ) specified by the user . The term “ Legendre ” of our algorithm comes from the fact that the theoretical support of our algorithm uses parameterization of tensors based on the Legendre transformation , which will be clarified in Section 3 . 2.1 Problem setup and notation . First we define the Tucker rank of tensors and formulate the problem of non-negative Tucker rank reduction . The Tucker rank of a dth-order tensor P ∈ RI1×···×Id is defined as a tuple ( Rank ( P ( 1 ) ) , . . . , Rank ( P ( d ) ) ) , where each P ( k ) ∈ RIk× ∏ m6=k Im is the mode-k expansion of the tensor P and Rank ( P ( k ) ) denotes the matrix rank of P ( k ) [ 11 , 15 , 39 ] . See Supplement for definition of the mode-k expansion . If the Tucker rank of a tensor P is ( r1 , . . . , rd ) , it can be always decomposed as P = ∑r1 i1=1 · · · ∑rd id=1 Gi1 , ... , ida ( 1 ) i1 ⊗ a ( 2 ) i2 ⊗ · · · ⊗ a ( d ) id with a tensor G ∈ Rr1×···×rd , called the core tensor of P , and vectors a ( k ) ik ∈ RIk , ik ∈ { 1 , . . . , rk } , for each k ∈ { 1 , . . . , d } , where ⊗ denotes the Kronecker product [ 24 ] . If every element of the Tucker rank is the same as each other element , it coincides with the CP rank . In this paper , we say that a tensor is rank-1 if its Tucker rank is ( 1 , . . . , 1 ) . The problem with non-negative Tucker rank reduction is approximating a given non-negative tensor by a non-negative lower Tucker rank tensor . We denote by [ d ] = { 1 , 2 , . . . , d } for a positive integer d and denote by Pa ( k ) : b ( k ) the subtensor obtained by fixing the range of kth index to only from a to b . 2.2 The LTR Algorithm . In LTR , we use the rank-1 approximation method that always finds the rank-1 tensor that minimizes the KL divergence from an input tensor [ 19 ] . The optimal rank-1 tensor P of P is given by P = λs ( 1 ) ⊗ s ( 2 ) ⊗ · · · ⊗ s ( d ) , ( 1 ) Algorithm 1 : input : Tensor P , target Tucker rank r = ( r1 , . . . , rd ) output : Rank reduced tensor Q LTR ( P , r ) ( I1 , . . . , Id ) ← the size of the input tensor P foreach k = 1 , . . . , d do Construct { c1 , . . . , crk } ⊆ [ Ik ] by random sampling from [ Ik ] without replacement , where we always assume that c1 = 1 and ci < ci+1 . foreach l = 1 , . . . , rk do if cl 6= cl+1 − 1 then Replace the subtensor P c ( k ) l : c ( k ) l+1−1 of P by its rank-1 approximation as P c ( k ) l : c ( k ) l+1−1 ← BESTRANK1 ( P c ( k ) l : c ( k ) l+1−1 ) Q ← P return Q BESTRANK1 ( P ) ( I1 , . . . , Id ) ← the size of the input tensor P foreach k = 1 , . . . , d do foreach ik = 1 , . . . , Ik do s ( k ) ik ←∑I1i1=1 · · ·∑Ik−1ik−1=1∑Ik+1ik+1=1 · · ·∑Idid=1 Pi1 , ... ik−1 , ik , ik+1 , ... , id λ← sum of all elements of P P ← λs ( 1 ) ⊗ s ( 2 ) ⊗ · · · ⊗ s ( d ) return P where each s ( k ) = ( s ( k ) 1 , . . . , s ( k ) Ik ) with k ∈ [ d ] is defined as s ( k ) ik = I1∑ i1=1 · · · Ik−1∑ ik−1=1 Ik+1∑ ik+1=1 · · · Id∑ id=1 Pi1 , ... , id and λ is the ( 1− d ) -th power of the sum of all elements of P . Now we introduce LTR , which iteratively applies the above rank-1 approximation to subtensors of a tensor P ∈ RI1×···×Id . When we reduce the Tucker rank of P to ( r1 , . . . , rd ) , LTR performs the following two steps for each k ∈ [ d ] : Step 1 : We construct C = { c1 , . . . , crk } ⊆ [ Ik ] by random sampling from [ Ik ] without replacement , where we always assume that c1 = 1 and cl < cl+1 for every l ∈ [ rk − 1 ] . Step 2 : For each l ∈ [ rk ] , if cl 6= cl+1 − 1 holds , we replace the subtensor Pc ( k ) l : c ( k ) l+1−1 of P by its rank-1 approximation obtained by Equation ( 1 ) . Step 1 of LTR requires O ( r1 + r2 + · · ·+ rd ) since we only need to sample integers from 1 , 2 , . . . , Ik for each k ∈ [ d ] using the Fisher-Yates method . Since the above procedure repeats the best rank-1 approximation at most r1r2 . . . rd times , the worst computational complexity of LTR is O ( r1r2 . . . rdI1I2 . . . Id ) . The choice of C in Step 1 is arbitrary , which means that another strategy can be used . For example , if we know that some parts of an input tensor are less important than others , we can directly choose these indices for C instead of random sampling to obtain a more accurate reconstructed tensor . We provide the algorithm of LTR in algorithmic format in Algorithm 1 .

Summary: The paper presents a low-rank tensor approximation algorithm for non-negative tensors. The algorithm is based on two observations. First, rank-1 approximation for tensors can be viewed as a mean-field approximation by treating each tensor as a probability distribution. Second, the authors provide a sufficient condition for distribution parameters to reduce Tucker ranks of tensors and, interestingly, this sufficient condition can be achieved by iterative application of the mean-field approximation. Since the mean-field approximation is always given as a closed formula, a fast low-rank approximation algorithm can be formulated without using a gradient method. The authors empirically demonstrate on several datasets that the algorithm is faster than the existing non-negative Tucker rank reduction methods ( in terms of achieving comparable approximations of given tensors).