The Complexity of Sparse Tensor PCA

√ n , we present a family of algorithms that smoothly interpolates between a simple polynomial-time algorithm and the exponential-time exhaustive search algorithm . For any 1 ≤ t ≤ k , our algorithms recovers the sparse vector for signal-tonoise ratio λ ≥ Õ ( √ t · ( k/t ) p/2 ) in time Õ ( np+t ) , capturing the state-of-the-art guarantees for the matrix settings ( in both the polynomial-time and sub-exponential time regimes ) . Our results naturally extend to the case of r distinct k-sparse signals with disjoint supports , with guarantees that are independent of the number of spikes . Even in the restricted case of sparse PCA , known algorithms only recover the sparse vectors for λ ≥ Õ ( k ·r ) while our algorithms require λ ≥ Õ ( k ) . Finally , by analyzing the low-degree likelihood ratio , we complement these algorithmic results with rigorous evidence illustrating the trade-offs between signal-to-noise ratio and running time . This lower bound captures the known lower bounds for both sparse PCA and tensor PCA . In this general model , we observe a more intricate three-way trade-off between the number of samples n , the sparsity k , and the tensor power p . 1 Introduction . Sparse tensor principal component analysis is a statistical primitive generalizing both sparse PCA2 and tensor PCA3 . We are given multi-linear measurements in the form of a tensor Y = W + λx⊗p ∈ ⊗pRn ( SSTM ) for a Gaussian noise tensorW ∈ ⊗pRn containing i.i.d . N ( 0 , 1 ) entries4 and signal-to-noise ratio λ > 0 . Our goal is to estimate the “ structured ” unit vector x ∈ Rn . The structure we enforce on x is sparsity : |supp ( x ) | ≤ k. The model can be extended to include multiple spikes in a natural way : Y = W + ∑r q=1 λqx ⊗p ( q ) , and even general order-p tensors : Y = W + ∑r q=1 λqX ( q ) for X ( q ) = x ( q,1 ) ⊗ · · · ⊗ x ( q , p ) ∈ ⊗pRn . In this introduction , we focus on the simplest single spike setting of SSTM . It is easy to see that sparse PCA corresponds to the setting with tensor order p = 2 . On the other hand , tensor PCA is captured by effectively removing the sparsity constraint : |supp ( x ) | ≤ n. In ∗Part of the work was done while the author was in ETH Zürich . 2Often in the literature , the terms sparse PCA and spiked covariance model refer to the sparse spiked Wishart model . However , here we consider the sparse spiked Wigner matrix model . 3Tensor PCA is also known as the spiked Wigner tensor model , or simply the spiked tensor model . 4Throughout the paper , we will write random variables in boldface . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recent years , two parallel lines of work focused respectively on sparse PCA [ JL09 , AW08 , BR13a , DM16 , HKP+17 , DKWB19 , HSV20 , dKNS20 ] and tensor PCA [ MR14 , HSS15 , MSS16 , HKP+17 , KWB19 , AMMN19 ] , however no result captures both settings . The appeal of the sparse spiked tensor model ( henceforth SSTM ) is that it allows one to study the computational and statistical aspects of these other fundamental statistical primitives in a unified framework , understanding the computational phenomena at play from a more general perspective . In this work , we investigate SSTM from both algorithmic and computational hardness perspectives . Our algorithm improves over known tensor algorithms whenever the signal vector is highly sparse . We also present a lower bound against low-degree polynomials which extends the known lower bounds for both sparse PCA and tensor PCA , leading to a more intricate understanding of how all three parameters ( n , k and p ) interact . 1.1 Related work . Disregarding computational efficiency , it is easy to see that optimal statistical guarantees can be achieved with a simple exhaustive search ( corresponding to the maximum likelihood estimator ) : find a k-sparse unit vector maximizing 〈Y , x⊗p〉 . This algorithm returns a k-sparse unit vector x̂ achieving constant squared correlation5 with the signal x as soon as λ & √ k · log ( np/k ) . That is , whenever λ & max‖x‖=1 , ‖x‖0=k〈W , x ⊗p〉 . Unfortunately , this approach runs in time exponential in k and takes super-polynomial time when p . k.6 As such , we assume p ≤ k from now on . Taking into account computational aspects , the picture changes . A good starting point to draw intuition for SSTM is the literature on sparse PCA and tensor PCA . We briefly outline some known results here . To simplify the discussion , we hide absolute constant multiplicative factors using O ( · ) , Ω ( · ) , . , and & , and hide multiplicative factors logarithmic in n using Õ ( · ) . 1.1.1 Sparse PCA ( Wigner noise ) . Sparse PCA with Wigner noise exhibits a sharp phase transition in the top eigenvalue of Y for λ ≥ √ n [ FP07 ] . In this strong signal regime , the top eigenvector7 v of Y correlates8 with x with high probability , thus the following spectral method achieves the same guarantees as the exhaustive search suggested above : compute a leading eigenvector of Y and restrict it to the top k largest entries in absolute value . Conversely , when λ < √ n , the top eigenvector of Y does not correlate with the signal x . In this weak signal regime , [ JL09 ] proposed a simple algorithm known as diagonal thresholding : compute the top eigenvector of the principal submatrix defined by the k largest diagonal entries of Y . This algorithm recovers the sparse direction when λ & Õ ( k ) , thus requiring almost an additional √ k factor when compared to inefficient algorithms . More refined polynomial-time algorithms ( low-degree polynomials [ dKNS20 ] , covariance thresholding [ DM16 ] and the basic SDP relaxation [ dGJL07 , dKNS20 ] ) only improve over diagonal thresholding by a logarithmic factor in the regime n1−o ( 1 ) . k2 . n. Interestingly , multiple results suggest that this information-computation gap is inherent to the sparse PCA problem [ BR13a , BR13b , DKWB19 , dKNS20 ] . Subexponential time algorithms and lower bounds have also been shown . For instance , [ DKWB19 , HSV20 ] presented smooth trade-offs between signal strength and running time.9 5One could also aim to find a unit vector with correlation approaching one or , in the restricted setting of x ∈ { 0 , ±1/ √ k } , aim to recover the support of x . At the coarseness of our discussion here , these goals could be considered mostly equivalent . 6Note that the problem input is of size np . So when p & k , exhaustive search takes nO ( p ) time which is polynomial in np . Thus , the interesting parameter regimes occur when p . k. 7By “ top eigenvector ” or “ leading eigenvector ” , we mean the eigenvector corresponding to “ largest ( in absolute value ) eigenvalue ” . 8More precisely , the vector consisting of the k largest ( in absolute value ) entries of v. 9Both works studied the single spike matrix setting . [ HSV20 ] only considers the Wishart noise model and thus its guarantees can not be compared to ours . [ DKWB19 ] studied both the Wishart and Wigner noise models . In the Wishart noise model setting , both [ HSV20 ] and [ DKWB19 ] observe the same tradeoff between running time and signal-to-noise ratio . In the Wigner noise model setting , our algorithm and the algorithm of [ DKWB19 ] offer the same smooth-trade off between running time and signal strength , up to universal constants . 1.1.2 Tensor PCA . In tensor settings , computing max‖x‖=1〈Y , x⊗p〉 is NP-hard already for p = 3 [ HL13 ] . For even tensor powers p , one can unfold the tensor Y into a np/2-by-np/2 matrix and solve for the top eigenvector [ MR14 ] . However , this approach is sub-optimal for odd tensor powers . For general tensor powers p , a successful strategy to tackle tensor PCA has been the use of semidefinite programming [ HSS15 , BGL16 , HSS19 ] . Spectral algorithms inspired by the insight of these convex relaxations have also been successfully applied to the problem [ SS17 ] . These methods succeed in recovering the single-spike x when λ & Õ ( np/4 ) , thus exhibiting a large gap when compared to exhaustive search algorithms . Matching lower bounds have been shown for constant degrees in the Sum-of-Squares hierarchy [ BGL16 , HKP+17 ] and through average case reductions [ BB20 ] . 1.1.3 Sparsity-exploiting algorithms and tensor algorithms . It is natural to ask how do the characteristics of sparse PCA and tensor PCA extend to the more general setting of SSTM . In particular , there are two main observations to be made . The first observation concerns the sharp computational transition that we see for k . √ n in sparse PCA . In these highly sparse settings , the top eigenvector of Y does not correlate with the signal x and so algorithms primarily based on spectral methods fail to recover it . Indeed , the best known guarantees are achieved through algorithms that crucially exploit the sparsity of the hidden signal . These algorithms require the signal strength to satisfy λ ≥ Õ ( √ k ) , with only logarithmic dependency on the ambient dimension . To exemplify this to an extreme , notice how the following algorithm can recover the support of xxT with the same guarantees as diagonal thresholding , essentially disregarding the matrix structure of the data : zero all but the k2 largest ( in absolute value ) entries of Y . A natural question to ask is whether a similar phenomenon may happen for higher order tensors . In the highly sparse settings where k . √ n , can we obtain better algorithms exploiting the sparsity of the hidden vector ? Recently , a partial answer appeared in [ LZ20 ] with a polynomial time algorithm recovering the hidden signal for λ ≥ Õ ( p · kp/2 ) , albeit with suboptimal dependency on the tensor order p. The second observation concerns the computational-statistical gap in the spiked tensor model . As p grows , the gap between efficient algorithms and exhaustive search widens with the polynomial time algorithms requiring signal strength λ ≥ Õ ( np/4 ) while exhaustive search succeeds when λ ≥ Õ ( √ n ) [ MR14 ] . The question here is : how strong is the dependency on p for efficient algorithms in sparse signal settings ? In this work , we investigate these questions in the high order tensors regime p ∈ ω ( 1 ) . We present a family of algorithms with a smooth trade-off between running time and signal-to-noise ratio . Even restricting to polynomial-time settings , our algorithms improve over previous results . Furthermore , through the lens of low-degree polynomials , we provide rigorous evidence of an exponential gap in the tensor order p between algorithms and lower bounds . Remark . The planted sparse densest sub-hypergraph model [ CPMB19 , BCPS20 , CPSB20 ] is closely related to SSTM . We discuss this model in Appendix E .

Summary: This paper studies the estimation of a rank-one (or more generally a low-rank) tensor $x^{\otimes p}$ ($x \in \mathbb{R}^n$ is assumed to have unit norm) given $Y = \lambda x^{\otimes p} + W$, where $\lambda > 0$ and $W$ is a Gaussian noise tensor. Here the signal vector $x$ is assumed to have only $k$ non-zero entries.  The main contribution of the paper is an algorithm that recovers with high probability the support of $x$ in time $O(n^{p+t})$ provided that $\lambda \geq O(\sqrt{t} (k/t)^{p/2})$. The number $t$ can be arbitrarily chosen between $1$ and $k$, illustrating a tradeoff between statistical constraints on the signal-to-noise ratio $\lambda$ and runtime constraints on the estimation algorithm. This generalizes to the 'tensor settings' previous results obtained about sparse PCA.  The authors also prove an information-theoretic lower bound for low-degree polynomials. More precisely, they prove that no test based on polynomial of degree at most $D$ can distinguish between the planted model $Y = \lambda x^{\otimes p} + W$ and the pure noise model $Y=W$, when $\lambda$ is smaller than some quantity depending on $D, n, k, p$. This bound generalizes previously obtained results for PCA and tensor PCA. This bound is tight for low-degree polynomials. However there is a gap with the  bound $\lambda \geq O(\sqrt{t} (k/t)^{p/2})$ needed for the algorithm that recovers the support.