Landscape analysis of an improved power method for tensor decomposition

1 Introduction . From applied and computational mathematics [ 27 ] to machine learning [ 36 ] and multivariate statistics [ 29 ] to many-body quantum systems [ 41 ] , high-dimensional data sets arise that are naturally organized into higher-order arrays . Frequently these arrays , known as hypermatrices or tensors , are decomposed into low-rank representations . In particular , the real symmetric CANDECOMP/PARAFAC ( CP ) decomposition [ 13 ] is often appropriate : T = K∑ i=1 λia ⊗m i . ( 1 ) Here , we are given T , a real symmetric tensor of size D×m . The goal is to expand T as a sum of K rank-1 terms , coming from scalar/vector pairs ( λi , ai ) ∈ R× RD . Importantly , the number of terms K must be minimal possible for the given tensor , in which case K is called the rank of the input T . When m > 2 and K = O ( Dm ) , CP decompositions are generically unique ( up to permutation and scaling ) by fundamental results in algebraic geometry [ 10 ] . An actionable interpretation [ 3 ] is that CP decomposition infers well-defined latent variables { ( λi , ai ) : i ∈ [ K ] } encoded by T . Indeed in learning applications , where symmetric tensors are formed from statistical moments ( higher-order covariances ) or multivariate derivatives ( higher-order Hessians ) , CP decomposition has enabled parameter estimation for mixtures of Gaussians [ 20 , 35 ] , generalized linear models [ 34 ] , shallow neural networks [ 19 , 24 , 42 ] , deeper networks [ 17 , 18 , 30 ] , hidden Markov models [ 5 ] , among others . Unfortunately , CP decomposition is NP-hard in the worst case [ 23 ] . In fact , it is believed to possess a computational-to-statistical gap [ 7 ] , with efficient algorithms expected to exist for random tensors 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . only up to rank K = O ( Dm/2 ) . For this reason , and due to its non-convexity and high-dimensionality , CP decomposition has served theoretically as a key testing ground for better understanding mysteries of non-convex optimization landscapes . To date , this focus has been on the non-convex program max x∈RD : ∥x∥2=1 〈 T , x⊗m 〉 . ( PM-P ) We label the problem PM-P , standing for Power Method Program , because projected gradient ascent applied to PM-P corresponds to the Shifted Symmetric Higher-Order Power Method of Kolda and Mayo [ 28 ] . Important analyses of PM-P include Ge and Ma ’ s [ 21 ] and the earlier [ 4 ] on overcomplete tensors , as well as [ 33 ] for low-rank tensors , and [ 31 ] which studied PM-P assuming the tensor components form a unit-norm tight frame with low incoherence . In this paper , we perform an analysis of the non-convex optimization landscape associated with the Subspace Power Method ( SPM ) for computing symmetric tensor decompositions . The first and third authors introduced SPM in [ 25 ] . This method is based on the following non-convex program : max x∈RD : ∥x∥2=1 FA ( x ) , ( SPM-P ) where FA ( x ) : = ∥PA ( x⊗n ) ∥2F , n : = ⌈m/2⌉ , A : = Span { a⊗n1 , . . . , a ⊗n K } , and PA : ( RD ) ⊗n → A is orthogonal projection with respect to Frobenius inner product . Note that SPM-P is a particular polynomial optimization problem of degree 2n on the unit sphere . There are at least two motivating reasons to study the optimization landscape of SPM-P. Firstly , it was observed in numerical experiments in [ 25 ] that the SPM algorithm is competitive within its applicable rank range of K = O ( D⌊m/2⌋ ) . It gave a roughly one-order of magnitude speed-up over the decomposition methods in [ 26 ] as implemented in Tensorlab [ 40 ] , while matching the numerical stability of FOOBI [ 14 ] . Thus SPM is a practical algorithm . Secondly , from a theory standpoint , the program SPM-P has certain desirable properties which PM-P lacks . Specifically for an input tensor T = ∑K i=1 λia ⊗m i with rank K ≲ D ⌊m/2⌋ and Zariski-generic1 { ( λi , ai ) } Ki=1 , SPM-P is such that : • Each component ±ai is exactly a global maximum , and there are no other global maxima . • The globally maximal value is known in advance to be exactly 1 . So the objective value gives a certificate for global optimality , and non-global critical points can be discarded . These properties were shown for SPM-P in [ 25 ] , but both fail for PM-P ( see Figure 1 ) . Thus , SPM-P is more relevant theoretically than PM-P as a test problem for non-convex CP decomposition . Prior theory . In [ 25 ] , it is proven that projected gradient ascent applied to SPM-P , initialized at almost all starting points with a constant explicitly-bounded step size , must converge to a secondorder critical point of SPM-P at a power rate or faster . However this left open the possible existence 1Zariski-generic means that the failure set can be described by the vanishing non-zero polynomials [ 22 ] , so in particular , has Lebesgue measure 0. of spurious second-order critical points , i.e. , second-order points with reasonably high objective value that are not global maxima ( unequal to , and possibly distant from , each CP component ±ai ) . Such critical points could pose trouble for the successful optimization of SPM-P . Furthermore all theory for SPM-P in [ 25 ] was restricted to the clean case : that is , when the input tensor T is exactly of a sufficiently low rank K. The analysis for PM-P in [ 21 , 31 , 33 ] also assume noiseless inputs . However , tensors arising in practice are invariably noisy due , e.g. , to sampling or measurement errors . Main contributions . We perform a landscape analysis of SPM-P by characterizing all second-order critical points , using suitable assumptions on a1 , . . . , aK . Under deterministic frame conditions on a1 , . . . aK , which are satisfied by mutually incoherent vectors , near-orthonormal systems , and random ensembles of size K = O ( D ) , Theorem 7 shows that all second-order critical points of SPM-P coincide exactly with±ai . Theorem 16 shows the same result for overcomplete random ensembles of size K = õ ( D⌊m/2⌋ ) , however requiring an additional superlevel set condition to exclude maximizers with vanishing objective values . Both results extend to noisy tensor decomposition , where SPM is applied to a perturbation T̂ ≈ T . In this setting , second-order critical points with objective values exceeding O ( ∥T̂ − T∥F ) are O ( ∥T̂ − T∥F ) -near to one of the components ±ai . We also show in Lemma 9 that spurious local maximizers ( with lower objective values ) do exist in the noisy case . The results imply a clear separation of the functional landscape between near-global maximizers , with objective values close to 1 , and spurious local maximizers with small objective value . Hence , the SPM objective can be used to validate a final iterate of projected gradient ascent in the noisy case . In Theorem 18 , we combine our landscape analysis with bounds on error propagation incurred during SPM ’ s deflation steps . This gives guarantees for end-to-end tensor decomposition using SPM . Lastly , we expose the relation between PM-P and SPM-P . Specifically , SPM-P can be expressed as PM-P with the appropriate insertion of the inverse of the Grammian ( Gn ) ij : = ⟨ai , aj⟩n . The resulting de-biasing effect on the local maximizers with respect to the components ±ai ( cf . Figure 1 ) is responsible for many advantages of SPM-P . Along the way , we state a conjecture about the minimal eigenvalue of Gn when ai are i.i.d . uniform on the sphere , which may be of independent interest . 2 Notation . Vectors and matrices . When x is a vector , ∥x∥p is the ℓp-norm ( p ∈ R≥1∪ { ∞ } ) . For x , y ∈ RD , the entrywise ( or Hadamard ) product is x⊙y ∈ RD , and the entrywise power is x⊙s : = x⊙. . .⊙x ∈ RD ( s ∈ N ) . When M is a matrix , ∥M∥2 is the spectral norm . If M is real symmetric , µj ( M ) is the eigenvalue of M that is the j-th largest in absolute value . We denote the identity by IdD ∈ RD×D . Tensors . A real tensor of length D and order m is an array of size D ×D × . . .×D ( m times ) of real numbers . Write T mD : = ( RD ) ⊗m ∼= RDm for the space of tensors of size D×m . Meanwhile , Sym ( T mD ) ⊆ T mD is the subspace of symmetric tensors ( i.e. , tensors unchanged by any permutation of indices ) . The Frobenius inner product and norm are denoted by ⟨· , ·⟩ and ∥·∥F , respectively . Given any linear subspace of tensors A ⊆ T mD , let PA : T mD → A denote the orthogonal projector onto A with respect to ⟨· , ·⟩ . In the case A = Sym ( T mD ) , the projector PSym ( T mD ) is the symmetrization operator , Sym : T mD → Sym ( T mD ) . Given T ∈ T m1 D and S ∈ T m2 D , the tensor ( or outer ) product is T ⊗ S ∈ T m1+m2D , defined by ( T ⊗ S ) i1 , ... , im1+m2 : = Ti1 , ... , im1Sim1+1 , ... , im2 . For T ∈ T mD and s ∈ N , the tensor power is T⊗s : = T ⊗ . . . ⊗ T ∈ T smD . For T ∈ T m1 D , S ∈ T m2D with m1 ≥ m2 , the contraction T · S ∈ T m1−m2 D is defined by ( T · S ) i1 , ... , im1−m2 : =∑ j1 , ... , jm2 Ti1 , ... , im1−m2 , j1 , ... , jm2Sj1 , ... , jm2 . Let Reshape ( T , [ d1 , . . . , dℓ ] ) be the function that re- shapes the tensor T to have dimensions d1 , . . . , dℓ , as in corresponding Matlab/NumPy commands . Other . The unit sphere in RD is SD−1 , and Unif ( SD−1 ) is the associated uniform probability distribution . Given a function f : RD → R , the Euclidean gradient and Hessian matrix at x ∈ RD are ∇f ( x ) ∈ RD and ∇2f ( x ) ∈ Sym ( T 2D ) . The Riemannian gradient and Hessian with respect to SD−1 at x ∈ SD−1 are ∇SD−1f ( x ) and ∇2SD−1f ( x ) ( see [ 1 ] ) . Write Span for linear span , [ K ] : = { 1 , . . . , K } , and |A| for the cardinality of a finite set A. Lastly , we use asymptotic notation freely .

Summary: This paper studies the non-convex landscape of the SPM-P optimization objective (in page 2) for tensor decomposition. Under both deterministic and random models of the components, the work shows that all second-order critical points coincide with the ground truth vectors (up to a sign difference) with different levels of overcompleteness. The result can also be generalized to the noisy cases under certain assumptions.