The Complexity of Sparse Tensor PCA

√ n , we present a family of algorithms that smoothly interpolates between a simple polynomial-time algorithm and the exponential-time exhaustive search algorithm . For any 1 ≤ t ≤ k , our algorithms recovers the sparse vector for signal-tonoise ratio λ ≥ Õ ( √ t · ( k/t ) p/2 ) in time Õ ( np+t ) , capturing the state-of-the-art guarantees for the matrix settings ( in both the polynomial-time and sub-exponential time regimes ) . Our results naturally extend to the case of r distinct k-sparse signals with disjoint supports , with guarantees that are independent of the number of spikes . Even in the restricted case of sparse PCA , known algorithms only recover the sparse vectors for λ ≥ Õ ( k ·r ) while our algorithms require λ ≥ Õ ( k ) . Finally , by analyzing the low-degree likelihood ratio , we complement these algorithmic results with rigorous evidence illustrating the trade-offs between signal-to-noise ratio and running time . This lower bound captures the known lower bounds for both sparse PCA and tensor PCA . In this general model , we observe a more intricate three-way trade-off between the number of samples n , the sparsity k , and the tensor power p . 1 Introduction . Sparse tensor principal component analysis is a statistical primitive generalizing both sparse PCA2 and tensor PCA3 . We are given multi-linear measurements in the form of a tensor Y = W + λx⊗p ∈ ⊗pRn ( SSTM ) for a Gaussian noise tensorW ∈ ⊗pRn containing i.i.d . N ( 0 , 1 ) entries4 and signal-to-noise ratio λ > 0 . Our goal is to estimate the “ structured ” unit vector x ∈ Rn . The structure we enforce on x is sparsity : |supp ( x ) | ≤ k. The model can be extended to include multiple spikes in a natural way : Y = W + ∑r q=1 λqx ⊗p ( q ) , and even general order-p tensors : Y = W + ∑r q=1 λqX ( q ) for X ( q ) = x ( q,1 ) ⊗ · · · ⊗ x ( q , p ) ∈ ⊗pRn . In this introduction , we focus on the simplest single spike setting of SSTM . It is easy to see that sparse PCA corresponds to the setting with tensor order p = 2 . On the other hand , tensor PCA is captured by effectively removing the sparsity constraint : |supp ( x ) | ≤ n. In ∗Part of the work was done while the author was in ETH Zürich . 2Often in the literature , the terms sparse PCA and spiked covariance model refer to the sparse spiked Wishart model . However , here we consider the sparse spiked Wigner matrix model . 3Tensor PCA is also known as the spiked Wigner tensor model , or simply the spiked tensor model . 4Throughout the paper , we will write random variables in boldface . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recent years , two parallel lines of work focused respectively on sparse PCA [ JL09 , AW08 , BR13a , DM16 , HKP+17 , DKWB19 , HSV20 , dKNS20 ] and tensor PCA [ MR14 , HSS15 , MSS16 , HKP+17 , KWB19 , AMMN19 ] , however no result captures both settings . The appeal of the sparse spiked tensor model ( henceforth SSTM ) is that it allows one to study the computational and statistical aspects of these other fundamental statistical primitives in a unified framework , understanding the computational phenomena at play from a more general perspective . In this work , we investigate SSTM from both algorithmic and computational hardness perspectives . Our algorithm improves over known tensor algorithms whenever the signal vector is highly sparse . We also present a lower bound against low-degree polynomials which extends the known lower bounds for both sparse PCA and tensor PCA , leading to a more intricate understanding of how all three parameters ( n , k and p ) interact . 1.1 Related work . Disregarding computational efficiency , it is easy to see that optimal statistical guarantees can be achieved with a simple exhaustive search ( corresponding to the maximum likelihood estimator ) : find a k-sparse unit vector maximizing 〈Y , x⊗p〉 . This algorithm returns a k-sparse unit vector x̂ achieving constant squared correlation5 with the signal x as soon as λ & √ k · log ( np/k ) . That is , whenever λ & max‖x‖=1 , ‖x‖0=k〈W , x ⊗p〉 . Unfortunately , this approach runs in time exponential in k and takes super-polynomial time when p . k.6 As such , we assume p ≤ k from now on . Taking into account computational aspects , the picture changes . A good starting point to draw intuition for SSTM is the literature on sparse PCA and tensor PCA . We briefly outline some known results here . To simplify the discussion , we hide absolute constant multiplicative factors using O ( · ) , Ω ( · ) , . , and & , and hide multiplicative factors logarithmic in n using Õ ( · ) . 1.1.1 Sparse PCA ( Wigner noise ) . Sparse PCA with Wigner noise exhibits a sharp phase transition in the top eigenvalue of Y for λ ≥ √ n [ FP07 ] . In this strong signal regime , the top eigenvector7 v of Y correlates8 with x with high probability , thus the following spectral method achieves the same guarantees as the exhaustive search suggested above : compute a leading eigenvector of Y and restrict it to the top k largest entries in absolute value . Conversely , when λ < √ n , the top eigenvector of Y does not correlate with the signal x . In this weak signal regime , [ JL09 ] proposed a simple algorithm known as diagonal thresholding : compute the top eigenvector of the principal submatrix defined by the k largest diagonal entries of Y . This algorithm recovers the sparse direction when λ & Õ ( k ) , thus requiring almost an additional √ k factor when compared to inefficient algorithms . More refined polynomial-time algorithms ( low-degree polynomials [ dKNS20 ] , covariance thresholding [ DM16 ] and the basic SDP relaxation [ dGJL07 , dKNS20 ] ) only improve over diagonal thresholding by a logarithmic factor in the regime n1−o ( 1 ) . k2 . n. Interestingly , multiple results suggest that this information-computation gap is inherent to the sparse PCA problem [ BR13a , BR13b , DKWB19 , dKNS20 ] . Subexponential time algorithms and lower bounds have also been shown . For instance , [ DKWB19 , HSV20 ] presented smooth trade-offs between signal strength and running time.9 5One could also aim to find a unit vector with correlation approaching one or , in the restricted setting of x ∈ { 0 , ±1/ √ k } , aim to recover the support of x . At the coarseness of our discussion here , these goals could be considered mostly equivalent . 6Note that the problem input is of size np . So when p & k , exhaustive search takes nO ( p ) time which is polynomial in np . Thus , the interesting parameter regimes occur when p . k. 7By “ top eigenvector ” or “ leading eigenvector ” , we mean the eigenvector corresponding to “ largest ( in absolute value ) eigenvalue ” . 8More precisely , the vector consisting of the k largest ( in absolute value ) entries of v. 9Both works studied the single spike matrix setting . [ HSV20 ] only considers the Wishart noise model and thus its guarantees can not be compared to ours . [ DKWB19 ] studied both the Wishart and Wigner noise models . In the Wishart noise model setting , both [ HSV20 ] and [ DKWB19 ] observe the same tradeoff between running time and signal-to-noise ratio . In the Wigner noise model setting , our algorithm and the algorithm of [ DKWB19 ] offer the same smooth-trade off between running time and signal strength , up to universal constants . 1.1.2 Tensor PCA . In tensor settings , computing max‖x‖=1〈Y , x⊗p〉 is NP-hard already for p = 3 [ HL13 ] . For even tensor powers p , one can unfold the tensor Y into a np/2-by-np/2 matrix and solve for the top eigenvector [ MR14 ] . However , this approach is sub-optimal for odd tensor powers . For general tensor powers p , a successful strategy to tackle tensor PCA has been the use of semidefinite programming [ HSS15 , BGL16 , HSS19 ] . Spectral algorithms inspired by the insight of these convex relaxations have also been successfully applied to the problem [ SS17 ] . These methods succeed in recovering the single-spike x when λ & Õ ( np/4 ) , thus exhibiting a large gap when compared to exhaustive search algorithms . Matching lower bounds have been shown for constant degrees in the Sum-of-Squares hierarchy [ BGL16 , HKP+17 ] and through average case reductions [ BB20 ] . 1.1.3 Sparsity-exploiting algorithms and tensor algorithms . It is natural to ask how do the characteristics of sparse PCA and tensor PCA extend to the more general setting of SSTM . In particular , there are two main observations to be made . The first observation concerns the sharp computational transition that we see for k . √ n in sparse PCA . In these highly sparse settings , the top eigenvector of Y does not correlate with the signal x and so algorithms primarily based on spectral methods fail to recover it . Indeed , the best known guarantees are achieved through algorithms that crucially exploit the sparsity of the hidden signal . These algorithms require the signal strength to satisfy λ ≥ Õ ( √ k ) , with only logarithmic dependency on the ambient dimension . To exemplify this to an extreme , notice how the following algorithm can recover the support of xxT with the same guarantees as diagonal thresholding , essentially disregarding the matrix structure of the data : zero all but the k2 largest ( in absolute value ) entries of Y . A natural question to ask is whether a similar phenomenon may happen for higher order tensors . In the highly sparse settings where k . √ n , can we obtain better algorithms exploiting the sparsity of the hidden vector ? Recently , a partial answer appeared in [ LZ20 ] with a polynomial time algorithm recovering the hidden signal for λ ≥ Õ ( p · kp/2 ) , albeit with suboptimal dependency on the tensor order p. The second observation concerns the computational-statistical gap in the spiked tensor model . As p grows , the gap between efficient algorithms and exhaustive search widens with the polynomial time algorithms requiring signal strength λ ≥ Õ ( np/4 ) while exhaustive search succeeds when λ ≥ Õ ( √ n ) [ MR14 ] . The question here is : how strong is the dependency on p for efficient algorithms in sparse signal settings ? In this work , we investigate these questions in the high order tensors regime p ∈ ω ( 1 ) . We present a family of algorithms with a smooth trade-off between running time and signal-to-noise ratio . Even restricting to polynomial-time settings , our algorithms improve over previous results . Furthermore , through the lens of low-degree polynomials , we provide rigorous evidence of an exponential gap in the tensor order p between algorithms and lower bounds . Remark . The planted sparse densest sub-hypergraph model [ CPMB19 , BCPS20 , CPSB20 ] is closely related to SSTM . We discuss this model in Appendix E .

Summary: This paper focuses on the tradeoff between computational complexity and conditions for recovery in a symmetric sparse tensor estimation problem. There has been a great deal of work on related problems recently in the case of matrices (order 2 tensors) and the case of tensors (without necessarily imposing sparsity constraints). The main achievability result (Theorem 1) provides sufficient condition for single spike model in terms of the ambient dimension, the order the tensors, the degree of sparsity, the signal strength, and an integer parameter the interpolates between a ````"low complexity" algorithm  "high complexity" algorithm (essentially brute force search). Similar results of for a multiple spike model are given in Theorem 2.   Necessary conditions for a certain testing problem are considered in Theorem 3 which provides necessary conditions such that a polynomial of bounded degree can "distinguish" between a planted model (with a signal of a given strength) and a null mull of only Gaussian noise. Here the criterion for distinguishability is a certain L2 test.  While similar results have been obtained in the literature, the authors argue that their results improve upon the existing ones in a number of ways.