A new framework for tensor PCA based on trace invariants

1 INTRODUCTION . Powerful computers and acquisition devices have made it possible to capture and store real-world multidimensional data . For practical applications ( Kolda & Bader ( 2009 ) ) , analyzing and organizing these high dimensional arrays ( formally called tensors ) lead to the well known curse of dimensionality ( Gao et al . ( 2017 ) , Suzuki ( 2019 ) ) . Thus , dimensionality reduction is frequently employed to transform a high-dimensional data set by projecting it into a lower dimensional space while retaining most of the information and underlying structure . One of these techniques is Principal Component Analysis ( PCA ) , which has made remarkable progress in a large number of areas thanks to its simplicity and adaptability ( Jolliffe & Cadima ( 2016 ) ; Seddik et al . ( 2019 ) ) . In the Tensor PCA , as introduced by Richard & Montanari ( 2014 ) , we consider a model where we attempt to detect and retrieve an unknown unit vector v0 from noise-corrupted multi-linear measurements put in the form of a tensor T. Using the notations found below , our model consists in : T = βv⊗k0 + Z , ( 1 ) with Z a pure Gaussian noise tensor of order k and dimension n with identically independent distributed ( iid ) standard Gaussian entries : Zi1 , i2 , ... , ik ∼ N ( 0 , 1 ) and β the signal-to-noise ratio . To solve this important problem , many methods have been proposed . However , practical applications require optimizable and parallelizable algorithms that are able to avoid the high computationally cost due to an unsatisfactory scalability of some of these methods . A summary of the time and space requirement of some existent methods can be found in Anandkumar et al . ( 2017 ) . One way to achieve this parallelizable algorithm is through methods based on tensor contractions ( Kim et al . ( 2018 ) ) which are extensions of the matrix product . These last years , tools based on tensor contractions have been developed by theoretical physicists where random tensors have emerged as a generalization of random matrices . In this paper , we investigate the algorithmic threshold of tensor PCA and some of its variants using the theoretical physics approach and we show that it leads to new insights and knowledge in tensor PCA . Tensor PCA and tensor decomposition ( the recovery of multiple spikes ) is motivated by the increasing number of problems in which it is crucial to exploit the tensorial structure ( Sidiropoulos et al . ( 2017 ) ) . Recently it was successfully used to address important problems in unsupervised learning ( learning latent variable models , in particular latent Dirichlet allocation Anandkumar et al . ( 2014 ) , Anandkumar et al . ( 2015 ) ) , supervised learning ( training of two-layer neural networks , Janzamin et al . ( 2015 ) ) and reinforcement learning ( Azizzadenesheli et al . ( 2016 ) ) . Related work Tensor PCA was introduced by Richard & Montanari ( 2014 ) where the authors suggested and analyzed different methods to recover the signal vector like matrix unfolding and power iteration . Since then , various other methods were proposed . Hopkins et al . ( 2015 ) introduced algorithms based on the sum of squares hierarchy with the first proven algorithmic threshold of nk/4 . However this class of algorithm generally requires high computing resources and relies on complex mathematical tools ( which makes its algorithmic optimization difficult ) . Other studied methods have been inspired by different perspectives like homotopy in Anandkumar et al . ( 2017 ) , statistical physics ( Arous et al . ( 2020 ) , Ros et al . ( 2019 ) , Wein et al . ( 2019 ) and Biroli et al . ( 2020 ) ) , quantum computing ( Hastings ( 2020 ) ) as well as statistical query ( Dudeja & Hsu ( 2020 ) ) . Recently , a fundamentally different set of mathematical tools that have been developed for tensors in the context of high energy physics have been used to approach the problem . They consist in trace invariants of degree d ∈ N , obtained by contracting pair of indices of d copies of the tensor T. They have been used in Evnin ( 2020 ) to study the highest eigenvalue of a real symmetric Gaussian tensor . Subsequently , Gurau ( 2020 ) provided a theoretical study on a function based on an infinite sum of these invariants . Their results suggest a transition phase for the highest eigenvalue of a tensor for β around n1/2 in a similar way to the BBP transition in the matrix case ( Baik et al . ( 2005 ) ) . Thus , this function allows the detection of a spike . However evaluating it involves computing an integral over a n-dimensional space , which may not be possible in a polynomial time . The contribution of this paper is the use of these invariant tools to build tractable algorithms with polynomial complexity . In contrast to Gurau ( 2020 ) , instead of using a sum of an infinite number of invariants , we select one trace invariant with convenient properties to build our algorithms . It lets us detect the presence of the signal linearly in time and with a space requirement in O ( 1 ) . Moreover , in order to recover the signal vector besides simply detecting it , we introduce new tools in the form of matrices associated to this specific invariant . Within this framework , we show as particular cases , that the two simpler graphs ( of degree two ) are similar to the tensor unfolding and the homotopy algorithms ( which is equivalent to average gradient descent ) . These two algorithms are the main practical ones known from the point of view of space and time requirement ( Anandkumar et al . ( 2017 ) provides a table comparison ) . Notations We use bold characters T , M , v for tensors , matrices and vectors and Tijk , Mij , vi for their components . [ p ] denotes the set { 1 , . . . , n } . A real k−th order tensor is of order k if it is a member of the tensor product of Rni , i ∈ [ k ] : T ∈ ⊗k i=1 Rni . It is symmetric if Ti1 ... ik = Tτ ( i1 ) ... τ ( ik ) ∀τ ∈ Sk where Sk is the symmetric group ( more details are provided in Appendix ? ? ) . For a vector v ∈ Rn , we use v⊗p ≡ v⊗ v⊗ · · · ⊗ v ∈ ⊗pRn to denote its p-th tensor power . 〈v , w〉 denotes the scalar product of v and w. Let ’ s define the operator norm , which is equivalent to the highest eigenvalue of a tensor of any order : ‖X‖op ≡ max { Xi1 , ... , ik ( w1 ) i1 . . . ( wk ) ik , ∀i ∈ { 1 , . . . , n } , ‖wi‖ ≤ 1 } The trace of A is denoted Tr ( A ) . We denote the expectation of a variable X by E ( X ) and its variance by σ ( X ) . We say that a function f is negligible compared to a positive function g and we write f = o ( g ) if limn→∞ f/g → 0 . Einstein summation convention It is important to keep in mind throughout the paper that we will follow the Einstein summation convention : when an index variable appears twice in a single term and is not otherwise defined , it implies summation of that term over all the values of the index . For example : TijkTijk ≡ ∑ ijk TijkTijk . It is a common convention when addressing tensor problems that helps to make the equations more comprehensible . 2 GENERAL FRAMEWORK FOR SIGNAL DETECTION AND RECOVERY . 2.1 WHAT DO WE USE TO STUDY THE SIGNAL ? . An important concept in problems involving matrices is the spectral theory . It refers to the study of eigenvalues and eigenvectors of a matrix . It is of fundamental importance in many areas . In machine learning , the matrix PCA computes the eigenvectors and eigenvalues of the covariance matrix of the features to perform a dimensional reduction while ensuring most of the key information is maintained . In this case , the eigenvalues is a very efficient tool to describe data variability . In the case of signal processing , eigenvalue can contain information about the intensity of the signal , while the eigenvector points out to its direction . Lastly , a more theoretical example involves quantum physics where the spectrum of the matrix operator is used to calculate the energy levels and the state associated . In all of these examples , an important property of the eigenvalues of a n-dimensional matrix M is its invariance under orthogonal transformations { M → OMO−1 , O ∈ O ( n ) } where O ( n ) is the n-dimensional orthogonal group ( i.e . the group of real matrices that satisfies OO > = In , which should not be confused with the computational complexity O ( n ) ) . Since these transformations essentially just rotate the basis to define the coordinate system , they must not affect intrinsic information like data variability , signal intensity or the energy of a system . The eigenvalues are able to capture some of these inherent information , but recovering the complete general information requires computing their respective eigenvectors ( for example to find the principal component , the direction of the signal or the physical state ) . There are more such invariants than eigenvalues . Another important set worth mentioning are the traces of the n first matrix powers Tr ( A ) , Tr ( A2 ) , . . . , Tr ( An ) . Obtaining them uses slightly different methods than eigenvalues , but they contain the same information since each set can be inferred from the other through some basic algebraic operations . On the basis of the matrix case , we expect that for a tensor T ∈ ⊗k i=1 Rni , tensor quantities that are invariant under orthogonal transformations ( Ta1j ... akj → O ( 1 ) a1jb 1 j . . . O ( k ) akj b k j Tb1j ... bkj for O ( i ) ∈ O ( ni ) ∀i ∈ [ k ] ) should capture similar intrinsic information like the intensity of the signal , and conceivably , there should be other objects related to these quantities that are able to indicate the direction of the signal . However , the concept of eigenvalue and eigenvector is ill defined in the tensor case and not practical giving that the number of eigenvalues is exponential with the dimension n ( Qi ( 2005 ) , Cartwright & Sturmfels ( 2013 ) ) and computing them is very complicated . In contrast , we have a very convenient generalization of the traces of the power matrices for the tensors that we call trace invariants . They have been extensively studied during the last years in the context of high energy physics and many important properties have been proven ( Gurau ( 2017 ) ) . We first give a more formal definition of trace invariants . Let T be a tensor whose entries are Ti1 , ... , ik . Let ’ s define a contraction of a pair of indices as setting them equal to each other and summing over them , as in calculating the trace of a matrix ( Aij → ∑n i=1 Aii ) . The trace invariants of the tensor T correspond to the different way to contract pairs of indices in a product of an even number of copies of T. The degree of the trace invariants consists in the number of copies of T contracted . For example , ∑ i1 , i2 , i3 Ti1i2i3Ti1i2i3 and ∑ i1 , i2 , i3 Ti1i2i2Ti1i3i3 are trace invariants of degree 2 . In the remainder of this paper , we will use the Einstein summation convention defined in the notation subsection . A trace invariant of degree d of a tensor T of order k admits a practical graphical representation as an edge colored graph G obtained by following two steps : we first draw d vertices representing the d different copies of T. The indices of each copy is represented by k half-edges with a different color for each index position as shown in Figure 1a . Then , when two different indices are contracted in the tensor invariant , we connect their corresponding half-edges in G. Reciprocally , to obtain the tensor invariant associated to a graph G with d vertices , we take d copies of T ( one for each vertex ) , we associate a color for each index position , and we contract the indices of the d copies of T following the coloring of the edges connecting the vertices . We denote this invariant IG ( T ) . Three important examples of trace invariants are : the melon diagram ( Figure 1b ) and the tadpole ( 1c ) . Avohou et al . ( 2020 ) provides a thorough study about the number of trace invariants for a given degree d. A very useful asset of these invariants is that we are able to compute their expectation for tensors whose components are Gaussian using simple combinatorial analysis ( Gurau ( 2017 ) ) .

Summary: This paper studies the detection and recovery problem in spiked tensor models in the form T = \beta v0^\otimes k + Z, where v0 is the underlying spike signal and Z is a Gaussian noise.  The authors claim that they propose a new framework to solve the problem, by looking at the trace invariants of tensors. The authors provide a detection algorithm (Algorithm 1) and a recovery algorithm (Algorithm 2), as well as the corresponding phases. The authors claim that: 1) they "build tractable algorithms with polynomial complexity", "a detection algorithm linear in time"; 2) the algorithms are very suitable for parallel architectures; 3) an improvement of the state of the art for the symmetric tensor PCA experimentally. The authors furthermore discuss the asymmetric case and the multiple spike case.