Title: Sharp Recovery Thresholds of Tensor PCA Spectral Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: tensor PCA

spectral algorithms

random matrix theory
Abstract: Many applications seek to recover low-rank approximations of noisy tensor data. We consider several practical and effective matricization strategies which construct specific matrices from such tensors and then apply spectral methods; the strategies include tensor unfolding, partial tracing, power iteration, and recursive unfolding. We settle the behaviors of unfolding and partial tracing, identifying sharp thresholds in signal-to-noise ratio above which the signal is partially recovered. In particular, we extend previous results to a much larger class of tensor shapes where axis lengths may be different. For power iteration and recursive unfolding, we prove that under conditions where previous algorithms partially recovery the signal, these methods achieve (asymptotically) exact recovery. Our analysis deploys random matrix theory to obtain sharp thresholds which elude perturbation and concentration bounds. Specifically, we rely upon recent disproportionate random matrix results, which describe sequences of matrices with diverging aspect ratio.

Sharp Recovery Thresholds of Tensor PCA Spectral Algorithms

Abstract

Many applications seek to recover low-rank approximations of noisy tensor data. We consider several practical and effective matricization strategies which construct specific matrices from such tensors and then apply spectral methods; the strategies include tensor unfolding, partial tracing , power iteration, and recursive unfolding. Wesettle the behaviors of unfolding and partial tracing, identifying sharp thresholds in signal-to-noise ratio above which the signal is partially recovered. In particular, we extend previous results to a much larger class of tensor shapes where axis lengths may be different. For power iteration and recursive unfolding, we prove that under conditions where previous algorithms partially recovery the signal, these methods achieve (asymptotically) exact recovery. Our analysis deploys random matrix theory to obtain sharp thresholds which elude perturbation and concentration bounds. Specifically, we rely upon recent disproportionate random matrix results, which describe sequences of matrices with diverging aspect ratio.

1 Introduction

Tensors-multi-way arrays-and tensor methods are fundamental to modern data analysis. Data given by three or more indices are increasingly the focus in diverse areas including signal and image processing [9, 19, 22], while high-order moments represented by tensors are of interest in problems such as community detection and learning latent variable models [1, 2, 8].

The spiked tensor model , introduced by Montanari and Richard [20], is a simple statistical model of tensor data with latent low-rank structure. Observed data X of dimensions n 1 × n 2 ×··· × n k are the sum of a low-rank tensor and noise:

where k ≥ 2 is the tensor order, λ is a signal strength, v j ∈ S n j -1 , j ∈ [ k ] , are unit vectors, and Z is a noise tensor. Noise entries are assumed to be independent standard Gaussians:

Estimation of v 1 , . . . , v k and the low-rank component of X generalizes to tensors the problem of low-rank matrix approximation-principal component analysis (PCA)-and is therefore known as tensor PCA.

This model reduces for k = 2 to the spiked matrix model , with maximum likelihood estimators of v 1 and v 2 given by the first left and right singular vectors ˆ v 1 and ˆ v 2 of X , respectively. For k ≥ 3 , however, computation of the maximum likelihood estimators of the low rank component amounts to solving an NP hard problem (Hillar and Lim [14]),

necessitating alternate, efficient algorithms.

1.1 Contributions

This paper studies matricization -based approaches, which convert tensors to matrices (by reshaping or by contracting constructions to be described) and then apply spectral methods. We assume the spiked tensor model in a high-dimensional asymptotic framework in which each array dimension n 1 , . . . , n k tends to infinity; we make no assumptions on their relative rates. While previous analyses of tensor PCA algorithms have assumed order k = 3 , supersymmetry v 1 = v 2 = · · · = v k , or hypercubical format n 1 = n 2 = · · · = n k , we permit tensors of diverse dimensions n j and unrelated v j .

In this setting, we fully analyze tensor unfolding or reshaping, a widespread technique [12, 19]. Additionally, we analyze the partial tracing approach of Hopkins et al. [16], discovering its asymptotic equivalence to unfolding in performance-an improvement over previous analysis. We identify sharp thresholds in signal-to-noise ratio above which these algorithms partially recover the signal, and provide exact formulas for their limiting performance. Finally, we study generalizations of the power iteration and recursive unfolding algorithms of [20]. Above the same thresholds characterizing partial recovery of unfolding and partial tracing, these algorithms achieve (asymptotically) exact signal recovery. In other words, for signal-to-noise ratios such that unfolding or partial tracing partially recover the signal, power iteration and recursive unfolding exactly recover the signal.

Our approach relies upon fundamental and penetrating random matrix theoretic (RMT) results. Precise analysis of tensor PCA algorithms, including pinpointing of phase transitions and sharp performance quantification as provided here, is not possible purely via more standard tools in theoretical machine learning, such as matrix concentration and vector perturbation bounds. Specifically, we rely upon recent disproportionate random matrix results (introduced in Section 1.3), which describe sequences of matrices with diverging aspect ratio.

This work concisely demonstrates in the context of tensor PCA the advantages of an RMT-backed approach. Indeed, application of spiked matrix results yields simple and elegant theorems and proofs which are easily read. For example, the analysis of the partial tracing in [16], involving challenging matrix concentration and perturbation calculations, is reduced to a few careful lines. Though our results are admittedly purely asymptotic, there are no hidden constants or logarithmic factors. Even for modest-sized tensors, simulations demonstrate close agreement with theory.

1.2 Assumptions and Notation

Without loss of generality, we assume λ ≥ 0 and n 1 ≤ n 2 ≤ ·· · ≤ n k , taking n 1 to be the 'fundamental' problem index; that is n j = n j ( n 1 ) , j ∈ [ k ] . We say an estimator ˆ v j partially recovers v j if almost surely,

and ˆ v j exactly recovers v j if |⟨ v j , ˆ v j ⟩| a.s. - - → 1 . In words, partial recovery demands a positive limit for the cosine similarity, while exact recovery requires asymptotically perfect cosine similarity.

We make the rank-one signal assumption in model (1) purely for expository efficiency; the spectral algorithms in this paper naturally generalize to rankr spiked tensors of the form

where r ≥ 1 is the rank and { v (1) j , . . . , v ( r ) j } ⊂ S n j -1 are orthonormal sets, j ∈ [ k ] . Such tensor decompositions are unique by Kruskal's theorem [18]. Where we estimate v (1) j by the first right singular vector of an appropriate matrix M j , v (2) j , . . . , v ( r ) j may be estimated by the subsequent right singular vectors, with analogous theoretical guarantees holding (that is, v ( i ) j is the right singular vector associated with the i -th largest singular value of M j ).

We denote by ⊗ both the tensor outer product and Kronecker product (the latter is simply a vectorization of the former); it is clear from context which is meant. Let × j denote multiplication between a tensor along the j -th axis and a vector of conformable dimension. The notation a ( n 1 ) ≲ b ( n 1 )

means a ( n 1 ) ≤ Cb ( n 1 ) for a constant C > 0 and all sufficiently large n 1 , and a ( n 1 ) ≍ b ( n 1 ) means a ( n 1 ) ≲ b ( n 1 ) ≲ a ( n 1 ) .

Finally, we introduce tensor slices . Let T ∈ R n 1 ×···× n k and fix ℓ ∈ [ k ] . The set { T i 1 ,...,i l : j ∈ [ l ] , i j ∈ [ n j ] } contains the slices of X of order k -ℓ . Slices satisfy T i 1 ,...,i ℓ ∈ ⊗ k j = ℓ +1 R n j and have entries

1.3 Spiked Matrix Model

The behavior of matricization-based algorithms fundamentally derives from spectral properties of the spiked matrix model,

This model is extensively studied in random matrix theory, particularly under proportional growth , where n 1 and n 2 are of comparable magnitude:

Under proportional growth, the sample covariance matrix XX ⊤ /n 2 does not converge to the identity (its expectation), and the leading left and right singular vectors ˆ v 1 and ˆ v 2 of X are inconsistent estimators of v 1 and v 2 , respectively. We highlight the following results of Benaych-Georges and Rao Nadakuditi [7], who establish formulas for the limiting misalignment of ˆ v 1 and ˆ v 2 :

Lemma 1.1. Let ˆ v 1 and ˆ v 2 denote the leading left and right singular vectors of X , respectively, and define

Under λ = τ (1 + o (1)) √ n 2 , where τ ≥ 0 is fixed, and γ n 1 → γ ∈ (0 , 1] ,

Recently, Ben Arous et al. [5] and Feldman [13] independently studied the spiked matrix model under disproportional growth , where γ n 1 → 0 or γ n 1 →∞ . Transposing X in case γ n 1 →∞ , we assume without loss of generality that γ n 1 → 0 . Phase transitions for the left and right singular vectors no longer coincide; v 1 is reliably estimated at signal strengths much weaker than λ ≳ √ n 2 . Drawing upon results in [5], [13], and [7], we have the following disproportionate analogs of Lemma 1.1:

Lemma 1.2. Let ˆ v 1 and ˆ v 2 denote the leading left and right singular vectors of X , respectively, and define

Under λ = τ (1 + o (1))( n 1 n 2 ) 1 / 4 , where τ ≥ 0 is fixed, and γ n 1 → 0 ,

while under λ = τ (1 + o (1)) √ n 2 ,

Limits (7) are corollaries of Theorems 2.9 and 2.10 of [7]. We note that [5] requires n 2 is polynomially bounded in n 1 . This assumption, however, is necessary only to establish non-asymptotic bounds; the almost-sure results in Lemma 1.2 hold as γ n 1 → 0 arbitrarily rapidly. While not explicitly stated in these references, it is easily verified that under proportional growth, lim inf λ/ √ n 2 > γ 1 / 4 implies partial recovery of v 1 and v 2 . Analogous statements hold for disproportional growth.

2 Tensor Unfolding

We study a general unfolding procedure that permits (1) tensors of general, unequal axis lengths and (2) unfolding along arbitrary sets of axes.

̸

Let N j = ∏ k ℓ = j n ℓ , j ∈ [ k ] , and for S ⊂ [ k ] , S = ∅ , let N ( S ) = ∏ ℓ ∈ S n ℓ . We define a map Mat S : R n 1 ×···× n k → R ( N 1 /N ( S )) × N ( S ) as follows: for indices i j ∈ [ n j ] , j ∈ [ k ] ,

and

When unfolding along a single axis, we write Mat j = Mat { j } . In this case, (8) reduces to

(taking N k +1 = 1 ) and b = i j . The unfolded matrix Mat j ( X ) is a spiked matrix with aspect ratio n 2 j /N 1 :

Similarly, Mat S ( X ) is a spiked matrix with aspect ratio N ( S ) 2 /N 1 :

We now review prior results on unfolding. For v 1 = · · · = v k , Montanari and Richard consider the unfolding S = { 1 , . . . , ⌊ k/ 2 ⌋} , estimating ⊗ ⌊ k/ 2 ⌋ v 1 by the first right singular vector of Mat S ( X ) . They prove that ⊗ ⌊ k/ 2 ⌋ v 1 is (partially) recovered for λ ≳ n ⌈ k/ 2 ⌉ / 2 1 , and conjecture sufficiency of λ ≳ n k/ 4 1 (note that these bounds differ only for odd k ). Theorem 5.8 of Hopkins et al. [15] completes this conjecture for k = 3 , establishing sufficiency of λ ≥ (1 + ε ) n 3 / 4 1 .

Complete analysis of unfolding-handling tensors of arbitrary order and asymmetric dimensionsrelies on results in Section 1.3, which reveal (1) necessary and sufficient thresholds for recovery and (2) the exact limiting performance of estimates. In addition, spiked matrix results yield the exact limiting cosine similarity of unfolding procedures, not obtained in [20] or [15]. The recent disproportional results summarized in Lemma 1.2 are crucial as unfoldings such as Mat j ( X ) , j ∈ [ k -1] , have limiting aspect ratio zero (recall that n 1 ≤ · · · ≤ n k ). Depending on the relative growth rates of n 1 , . . . , n k , unfoldings such as Mat k ( X ) may fall under disproportional 'tall' growth ( n 2 k /N 1 → 0 ), proportional growth ( n 2 k /N 1 ≍ 1 ), or disproportional 'wide' growth ( n 2 k /N 1 →∞ ) . Our analysis is similar to that of Ben Arous et al. [5], which permits arbitrary order k yet assumes n 1 = · · · = n k .

Theorem 2.1. Let S ⊂ [ k ] . If N ( S ) 2 /N 1 → 0 , the first right singular vector w of the unfolding Mat S ( X ) partially recovers ⊗ j ∈ S v j if and only if lim inf λ/N 1 / 4 1 > 1 . In particular, for λ = τ (1 + o (1)) N 1 / 4 1 with τ ≥ 0 fixed,

If N ( S ) 2 /N 1 ≍ 1 , ⊗ j ∈ S v j is partially recovered if and only if lim inf λ/N 1 / 4 1 > 1 . For γ n 1 → γ and λ = τ (1 + o (1)) N 1 / 4 1 ,

Finally, if N ( S ) 2 /N 1 →∞ , ⊗ j ∈ S v j is partially recovered if and only if lim inf λ/ √ N ( S ) > 0 . For λ = τ (1 + o (1)) √ N ( S ) ,

All proofs are deferred to the supplement. As the recovery threshold is identical across all unfoldings S such that N ( S ) 2 /N 1 ≲ 1 , we propose the following simple algorithm to estimate v j , which may be iterated for j ∈ [ k ] :

Algorithm 1 Tensor unfolding

Input: X,j return first right singular vector of Mat j ( X )

Tensor unfolding

1.0

n1=400

0.8

<2

n1=1200

()

cosine similarity

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Tensor unfolding, (n1,n2,n3) = (50,50,10000)

Tensor unfolding, (n1, n2, n3) = (50, 50,20000)

1.0

1.0

0.8

0.8

#

cosine similarity

0.6

0.6

,)1

[<,01)1

0.4

0.4

<v2, 02)1

[<, 02)1

1<03, 03)1

[(v3, 03)1

0.2

C(X/N1/4)

0.2

C(X/N1/4)

c(入/√n3,N/n²)

c(X/√n3,N/n²)

0.0

0.0

50

100

150

200

250

50

100

150

200

250

signalstrength入

signal strength X

3 Partial Tracing

Hopkins et al. [16] propose the partial tracing method for tensors of order k = 3 as an alternative to unfolding. Their approach generalizes to orders k ≥ 3 , although it inherently relies on supersymmetry

of the signal, v 1 = · · · = v k . The partial tracing operator Tr k : ⊗ k R n ↦→ R n × n constructs a matrix by linearly combining tensor slices with partial trace weights:

This operation of reducing the order of a tensor by linear combinations of slices is called contraction in tensor analysis.

Algorithm 2 Partial tracing

Input: X

return first right singular vector of Tr k ( X )

For k = 3 , the runtime of Algorithm 2 is O ( n 3 ) , while that of unfolding is O ( n 3 log n ) (see Table 3 of [16]). Hopkins et al. established a bound on the recovery threshold of partial tracing which is worse than that of unfolding by a logarithmic factor: λ ≳ n 3 / 4 (log n ) 1 / 2 . We eliminate the logarithmic factor here: while delivering improvements in runtime, Algorithm 2 is asymptotically equivalent to unfolding in recovery performance.

Theorem 3.1. The first right singular vector ˆ v 1 of the partial trace matrix Tr k ( X ) partially recovers v 1 if and only if lim inf λ/n k/ 4 1 > 1 . For λ = τ (1 + o (1)) n k/ 4 1 , where τ is fixed,

Under n 1 = · · · = n k , the behavior of unfolding is given by (12), matching (16) exactly. Thus, unfolding and partial tracing are asymptotically equivalent in performance.

Partial tracing

1.0

n1=400

0.8

n1=1200

()

cosine similarity |

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

4 Exact Recovery

Montanari and Richard [20] consider a two-step recursive unfolding procedure that remarkably exactly recovers v 1 above the threshold of unfolding, assuming even order k , hypercubical shape n 1 = · · · = n k , and a supersymmetric signal, v 1 = · · · = v k . In this section, we prove that exact recovery is possible for tensors of arbitrary axis lengths. We consider generalizations of the power iteration and recursive unfolding algorithms of [20] (initialized via tensor unfolding). Tensor power iteration is previously studied in [17] (assuming a supersymmetric signal) and [21] (assuming

n 1 ≍ n 2 ≍ · · · ≍ n k ), though these works assume the initial iterate is independent of X , precluding initialization via unfolding.

In our more general setting, we find that unfolding-initialized power iteration achieves exact recovery above the partial recovery threshold of unfolding, established in Corollary 2.1.1 (assuming n 2 k /N 1 → 0 ). Recursive unfolding exactly recovers v 1 , . . . , v k -1 with no limitation on the growth rate of n k . Power iteration, which requires initial 'warm' estimates of v 1 , . . . , v k , may fail when n 2 k and N 1 are comparable. Recursive unfolding, on the other hand, does not rely upon an initial estimate of v k to recover v 1 , . . . v k -1 exactly.

Algorithm 3 Power iteration

We prove that exact recovery is achieved in a single iteration of the outer loop. In practice, iterating until estimates converge is beneficial.

Theorem 4.1. If lim inf λ/N 1 / 4 1 > 1 and n 2 k /N 1 → 0 , Algorithm 3 recovers v 1 , . . . , v k exactly: denoting by ˆ v 1 , . . . , ˆ v k the estimates after a single iteration of the outer loop,

Estimating the signal strength λ by ˆ λ = ⟨ X, ˆ v 1 ⊗··· ⊗ ˆ v k ⟩ , we have

We next consider a recursive unfolding procedure. As stated above, recursive unfolding is guaranteed to recover v 1 , . . . , v k -1 exactly if lim inf λ/N 1 / 4 1 > 1 ; no bound is needed on the growth rate of n k as in Theorem 4.1. In practice, it is beneficial to iterate Algorithm 4.

Algorithm 4 Recursive unfolding

Input: X , initial estimates ˆ v 1 , . . . , ˆ v k from Algorithm 1 for j from 2 to k : ˆ v j ← first right singular vector of Mat j -1 ( X × 1 ˆ v 1 ) end for ˆ v 1 ← first right singular vector of Mat 1 ( X × 2 ˆ v 2 ) return ˆ v 1 , . . . , ˆ v k

Theorem 4.2. Under lim inf λ/N 1 / 4 1 > 1 , Algorithm 4 recovers v 1 , . . . , v k -1 exactly. Moreover, v k is recovered exactly if n 2 k /N 1 ≲ 1 .

The analysis of Algorithm 4 entails careful application of spiked matrix model results. By the linearity of the unfolding operator,

Observe that Z × 1 ˆ v 1 is a reshaping of the vector Mat 1 ( Z )ˆ v 1 . As ˆ v 1 and Z are dependent, the second term on the right-hand side is not a matrix of i.i.d. entries. Despite dependencies, we claim that appropriately scaled, Mat 1 ( Z )ˆ v 1 is distributed as Gaussian noise, in which case exact recovery thresholds are a consequence of spiked matrix model results of Section 1.3 applied to (19).

Below, we display simulation results for power iteration and recursive unfolding and compare to tensor unfolding and partial tracing.

Power iteration

1.0

n=400

n=800

0.8

n=1200

1( > 1)

cosine similarity

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

signal strength T=入/n²

3/4

Power iteration,(n1,n2,n3)=(50,50,10000)

Power iteration,(n1,n2,n3)=(50,50,20000)

1.0

1.0

2

0.8

0.8

#

cosine similarity

0.6

0.6

0.4

0.4

[(0,01)1

[(,0)

0.2

0.2

{<02, 02)1

[(,2)

1<v3, 03)1

1<v3, 03)1

0.0

0.0

50

100

150

200

250

50

100

150

200

250

signalstrength入

signalstrength入

Comparison of algorithms, n1 = 1200

1.0

unfolding

partial tracing

0.8

poweriteration

()

cosine similarity

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Recursive unfolding,(n1,n2,n3)=(50,50,10000)

Recursive unfolding, (n1,n2,n3)= (50,50,50000)

1.0

1.0

[(01, 01)1

(<02, 02)1

2

0.8

0.8

[(U3,03)

1,

cosine similarity

0.6

0.6

0.4

0.4

[<01,01)1

0.2

0.2

(<02, 02)1

1<03, 03)1

0.0

0.0

50

100

150

200

250

50

100

150

200

250

signalstrength入

signalstrength入

References

A. Anandkumar, R. Ge, D. Hsu, and S. M. Kakade. A Tensor Approach to Learning Mixed Membership Community Models. Journal of Machine Learning Research , 1:2239-2312. 2014.

A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky. Tensor Decompositions for Learning Latent Variable Models. Journal of Machine Learning Research , 15:2773-2832, 2014.

M. Bayati and A. Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory , 57(2):764-785, 2011.

G. Ben Arous, R. Gheissari, and A. Jagannath. Algorithmic thresholds for tensor PCA. Annals of Probability , 48(4):2052-2087, 2020.

[5]G. Ben Arous, D. Z. Huang, and J. Huang. Long Random Matrices and Tensor Unfolding. arXiv preprint arXiv2110.1021, 2021.

G. Ben Arous, S. Mei, A. Montanari, and M. Nica. The landscape of the spiked tensor model. Communications on Pure and Applied Mathematics , 72:2282-2330, 2019.

F. Benaych-Georges and R. R. Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis , 111:120-135, 2012.

A. R. Benson, D. F. Gleich, and J. Leskovec. Tensor spectral clustering for partitioning higher-order network structures. Proceedings of the 2015 SIAM International Conference on Data Mining , 118-126, 2015.

X. Bi, X. Tang, Y. Yuan, Y. Zhang, A. Qu. Tensors in Statistics. Annual Review of Statistics and Its Application , 8:345-368, 2021.

A. Bloemendal, L. Erdos, A. Knowles, H. T. Yau, and J. Yin. Isotropic local laws for sample covariance and generalized Wigner matrices. Electronic Journal of Probability , 19(33):1-53, 2014.

B. B. Chen and G. M. Pan. Convergence of the largest eigenvalue of normalized sample covariance matrices when p and n both tend to infinity with their ratio converging to zero. Bernoulli , 18(4):1405-1420, 2012.

L. De Lathauwer, B. De Moor, and J. Vandewalle. A Multilinear Singular Value Decomposition. SIAM Journal on Matrix Analysis and Applications , 21(4):1253-1278, 2000.

M. J. Feldman. Spiked Singular Values and Vectors under Extreme Aspect Ratios. Journal of Multivariate Analysis , 196:105187, 2023.

C. J. Hillar and L. Lim. Most tensor problems are NP-hard. Journal of the ACM , 60(6):1-39, 2013.

S. B. Hopkins, J. Shi, and D. Steurer. Tensor principal component analysis via sum-of-square proofs. Conference on Learning Theory , 956-1006, 2015.

S. B. Hopkins, T. Schramm, J. Shi, and D. Steurer. Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors. Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing , 178-191, 2016.

J. Huang, D. Z. Huang, Q. Yang, and G. Cheng. Power Iteration for Tensor PCA. Journl of Machine Learning Research , 23:1-47, 2022.

J. B. Kruskal, Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics, Linear Algebra and Applications 18(2):95-138, 1977.

H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos. A survey of multilinear subspace learning for tensor data. Pattern Recognition , 44(7):1540-1551, 2011.

A. Montanari and E. Richard. A statistical model for tensor PCA. Proceedings of the 27th International Conference on Neural Information Processing Systems , 2:2897-2905, 2014.

[21] M. E. A. Seddik, M. Guillaud, and R. Couillet. When random tensors meet random matrices. arXiv preprint arXiv:2112.12348, 2021.

[22] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos. Tensor Decomposition for Signal Processing and Machine Learning. IEEE Transactions on Signal Processing , 65(13):3551-3582, 2017.

R. Tomioka and T. Suzuki. Spectral norm of random tensors, 2014. arXiv:1407.1870.

[24] M. Wainwright. High-Dimensional Statistics . Cambridge, 2019.

Y. Yu, T. Wang, and R. J. Samworth. A useful variant of the Davis-Kahan theorem for statisticians. Biometrika , 102:315-323, 2015.

Appendix

proof of Theorem (2.1). By (11), Mat S ( X ) is a spiked matrix with aspect ratio N ( S ) 2 /N 1 and signal strength λ (as ⊗ j ∈ S v j is unit norm). There are four cases to consider:

Let N ( S ) 2 /N 1 → 0 ; by the first point of (6), the recovery threshold lies at ( N 1 /N ( S ) · N ( S )) 1 / 4 = N 1 / 4 1 .

Let N ( S ) 2 /N 1 → γ ∈ (0 , 1] ; by the first point of (5), partial recovery occurs for lim inf λ/ √ N 1 /N ( S ) > γ 1 / 4 . Equivalently, lim inf λ/N 1 / 4 1 > 1 . Under λ = τ (1 + o (1)) N 1 / 4 1 , lim λ/ √ N 1 /N ( S ) = τγ 1 / 4 , implying |⟨⊗ j ∈ S v j , w ⟩| 2 a.s. - - → c 2 ( τγ 1 / 4 , γ ) .

Let N ( S ) 2 /N 1 → γ ∈ (1 , ∞ ] ; by the second point of (5), partial recovery occurs for lim inf λ/ √ N ( S ) > γ -1 / 4 . Equivalently, lim inf λ/N 1 / 4 1 > 1 . Under λ = τ (1 + o (1)) N 1 / 4 1 , lim λ/ √ N ( S ) = τγ -1 / 4 , implying |⟨⊗ j ∈ S v j , w ⟩| 2 a.s. - - → c 2 ( τγ -1 / 4 · γ 1 / 2 , γ ) = c 2 ( τγ 1 / 4 , γ ) .

Let N ( S ) 2 /N 1 →∞ ; by the second point of (7), the recovery threshold lies at √ N ( S ) .

Proof of Theorem 3.1. We provide proof for k = 3 ; the proof for higher orders is similar and omitted. For notational simplicity, we suppress the subscripts of n 1 and v 1 .

Let w i = tr ( Z i ) , i ∈ [ n ] , and w = ( w 1 , . . . , w n ) ⊤ . Expanding the partial trace matrix,

where ˜ Z is an independent copy of Z .

Let M = ∥ λv + w ∥ -1 2 ∑ n i =1 ( λv i + w i )( Z i -diag ( Z i -˜ Z i )) . As w i and Z i -diag ( Z i -˜ Z i ) are independent and the Gaussian distribution is rotationally invariant, M d = Z 1 . Hence, we have

where α = ∥ λv + w ∥ -1 2 ( λ 2 + λ ⟨ v, w ⟩ ) . The partial trace matrix is thus proportional to a perturbation of a spiked matrix with aspect ratio γ = 1 .

Let E denote the third term on the right-hand side of (21); we shall prove n -1 / 2 ∥E∥ 2 a.s. - - → 0 . Denoting u = ( λv + w ) / ∥ λv + w ∥ 2 , the diagonal entries of ∑ n i =1 u i diag ( ˜ Z i ) are i.i.d. Gaussians with variance one, implying ∥ ∑ n i =1 u i diag ( ˜ Z i ) ∥ 2 ≲ √ log n , almost surely. Similarly, since w ∼ N (0 , nI n ) and ∥ λv + w ∥ 2 = Θ a.s. ( λ + n ) ,

To bound the remaining term of E , let Z ∈ R n × n denote the matrix with entries Z ij = Z ijj , i, j ∈ [ n ] , in which case we may write

where 1 n is the lengthn vector of ones. As e ⊤ j Z ⊤ Z e j ∼ χ 2 n and

standard bounds such as (2.19) in [24] yield

where c, C > 0 are constants. As the right-hand side is summable, the Borel-Cantelli lemma implies

almost surely. Collecting the above bounds, we have that n -1 / 2 ∥E∥ 2 a.s. - - → 0 .

By Weyl's inequality, the limiting spectral distribution of n -1 / 2 ( M + E ) equals that of n -1 / 2 M , the quarter circle law. The limits (5) from Lemma 1.1 therefore apply to αvv ⊤ + M + E as well (see [7]). Basic calculations yield (1) lim inf α/ √ n > 1 if and only if lim inf λ/n 3 / 4 > 1 , and (2) under λ = τ (1 + o (1)) n 3 / 4 , α = τ 2 (1 + o a.s. (1)) √ n . Therefore, ˆ v partially recovers v if and only if lim inf λ/n 3 / 4 > 1 , and under λ = τ (1 + o (1)) n 3 / 4 ,

completing the proof.

Proof of Theorem 4.1. By the linearity of the operators × 1 , . . . , × k ,

Assume that (17) holds for ℓ ∈ [ j -1] . For ℓ ∈ { j +1 , . . . , k } , Corollary 2.1.1 implies |⟨ v ℓ , ˆ v ℓ ⟩| is bounded away from zero. The first term on the right-hand side of (28) is therefore Θ a.s. ( λ ) . The second term is bounded by the spectral norm of Z : using Theorem 1 of [23],

almost surely. Thus, as ( √ n k /λ ) 4 ≲ n 2 k /N 1 → 0 ,

from which (17) follows inductively.

Equation (18) follows from (17) and (29), which imply ∥ v 1 ⊗···⊗ v k -ˆ v 1 ⊗···⊗ ˆ v k ∥ F a.s. - - → 0 and

Proof of Theorem 4.2. We shall prove v 2 , . . . , v k are recovered exactly; proofs for the first and last axes are similar and omitted. By the linearity of the unfolding operator,

Observe that Z × 1 ˆ v 1 is a reshaping of the vector Mat 1 ( Z )ˆ v 1 . As ˆ v 1 and Z are dependent, the second term on the right-hand side is not a matrix of i.i.d. entries. Despite dependencies, we claim that appropriately scaled, Mat 1 ( Z )ˆ v 1 is Gaussian noise, in which case exact recovery thresholds are a consequence of spiked matrix model results of Section 1.3 applied to (32).

By definition, ˆ v 1 is the first eigenvector of the symmetric matrix

where E is a rank-two matrix given by

Let ˜ v 1 denote the first eigenvector of Mat 1 ( X ) ⊤ Mat 1 ( X ) -E ; without loss of generality, we assume ˆ v ⊤ 1 ˜ v 1 ≥ 0 . Since Mat 1 ( Z ) ⊤ ( ⊗ ℓ ∈ [ k ] \{ 1 } v ℓ ) ∼ N (0 , I n 1 ) , we have ∥E∥ 2 ≲ λ √ n 1 almost surely. By Theorem 1.1 of [13], the spectral gap of Mat 1 ( X ) ⊤ Mat 1 ( X ) is Θ( λ 2 ) . Thus, using the Davis-Kahan theorem (see Corollary 3 of [25]), we have

Let Mat 1 ( Z ) = U Λ V ⊤ be a singular value decomposition. As Z contains i.i.d. Gaussian entries, (1) U , Λ , and V are independent, (2) U and V are Haar-distributed. Moreover, as ˜ v 1 is the first eigenvector of λ 2 v 1 v ⊤ 1 + Mat 1 ( Z ) ⊤ Mat 1 ( Z ) = λ 2 v 1 v ⊤ 1 + V Λ 2 V ⊤ , ˜ v 1 is independent of U . Thus, U Λ V ⊤ ˜ v 1 / ∥ Λ V ⊤ ˜ v 1 ∥ 2 is uniform on S N 2 -1 . Generating ξ ∼ χ 2 N 2 independent of Z , it follows that

Defining the constant α = ξ/ ( √ N 2 ∥ Λ V ⊤ ˜ v 1 ∥ 2 ) , we deduce that αZ × 1 ˜ v 1 (which is a reshaping of the left-hand side of (35)) is distributed as a tensor with i.i.d. Gaussian entries-despite dependencies between Z and ˜ v 1 . Additionally, since Λ 11 / √ N 2 a.s. - - → 1 and Λ n 1 n 1 / √ N 2 a.s. - - → 1 , we have ∥ Λ V ⊤ ˜ v 1 ∥ 2 / √ N 2 a.s. - - → 1 and α a.s. - - → 1 .

Thus, we conclude that

where ˜ Z ∈ R n j × ( N 2 /n j ) contains i.i.d. Gaussian entries, enabling us to apply spiked matrix results. Let ˜ v j denote the first right singular vector of Mat j -1 ( X × 1 ˜ v 1 ) (equivalently, that of α Mat j -1 ( X × 1 ˜ v 1 ) ); without loss of generality, we assume ˆ v ⊤ 1 ˜ v 1 ≥ 0 . In particular, since |⟨ v 1 , ˜ v 1 ⟩| ≍ |⟨ v 1 , ˆ v 1 ⟩| ≍ 1 by Corollary 2.1.1 and (34), Lemma 1.2 implies |⟨ v j , ˜ v j ⟩| a.s. - - → 1 (we have λ ≫ ( n j · N 2 /n j ) 1 / 4 ) .

It therefore suffices to prove that ∥ ˆ v j -˜ v j ∥ 2 a.s. - - → 0 . By Theorem 2.3 of [5] or Theorem 1.1 of [13] and Cauchy's interlacing inequality, the spectral gap of Mat j -1 ( X × 1 ˜ v 1 ) ⊤ Mat j -1 ( X × 1 ˜ v 1 ) is Θ( λ 2 ) . Let Z denote the reshaping of Z with dimensions n 1 × n j × N 2 /n j and slices Mat j -1 ( Z i ) , i ∈ [ n 1 ] . Using the bound

and Theorem 1 of [23],

almost surely. Thus, using the Davis-Kahan theorem (Theorem 4 of [25]), (34), and (37), we have

completing the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the (Gaussian, nonsymmetric, rank-1) spiked tensor model introduced in Montanari and Richard.  It gives sharp thresholds for various matricization techniques (unfolding, partial traces, successive contraction).  The statements about random tensors are reduced to known results about left and right singular vectors of spiked non-symmetric random matrices.  The proofs of the new statements are a few lines long.  Some numerical results back up the performance statement about successive contraction.\n\nSTRENGTHS:\nThe presentation is generally clear and concise.  The proofs seem reasonably detailed given the content matter.  Some of the results appear novel and improve over substantially longer published work.  Highlights include\n1) Improving an analysis in Hopkins et al.\n2) Generalizing the successive contraction method of Montanari and Richard with excellent performance properties.\n\nWEAKNESSES:\nThe new theoretical content in this paper is very slight: the random matrix theory of spiked Gaussian matrices is extremely well-developed. Once it is noticed that the unfolded tensors satisfy the hypotheses of these theorems, the proof is immediate.\n\nThere is no further elaboration beyond this:\n1) the paper does not show how this fits into a larger picture of non-random-matrix-theory approaches.\n2) the paper does not attempt to expand interesting unexplored aspects which don't follow from textbook random matrix theory (the best hint of this are the numerics that appear for tau=1 at the end).\n3) the new technique (in the abstract \"Finally, we introduce an iterative algorithm based on successive contractions of the tensor\") is rather a generalization of a technique of Montanari and Richard.  The analysis given is in the phase of perfect recovery, exploiting some of the specific structure of the spiked model.\n\nThis is a reasonable topic for a purely theoretical NeurIPS contribution, but there is not enough new material here to warrant publication.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies recovery of planted signals from noisy tensors. \nThe model is the prototypical tensor PCA and the emphasis is that (i) the planted low rank tensors (in particular their dimensions) can be different along different modes; (ii) the joint scaling of different dimensions can deviate from the classical proportional regime. \nThe paper studies a few old and new natural algorithms and obtain sharp performance guarantees (specifically the overlap) for almost all of them.\n\nSTRENGTHS:\nBoth the techniques and the results of this paper are neat. \nIt properly combines recent (and some not-so-recent) results in random matrix theory (RMT) and high-dimensional statistics and extend them to a satisfactory extent. \nOnce some prior results are properly organized, the proofs of the present results become rather straightforward. \nBut for ICML, I think this is enough. \nAlso, I thought about a subset of the questions studied in this paper before (perhaps before some of the recent RMT results were available) and at that time it was not clear how these results can be proved.\n\nWEAKNESSES:\nOne important aspect of tensor PCA that the authors seem to ignore by large is the potential existence of information-computation gap. \nThe authors may want to check papers of Sam Hopkins, David Steurer, Andrea Montanari, Guy Bresler, Alex Wein (listed in random order) and many others. \nThey provide evidence of average case hardness from the perspectives of SoS, low-degree polynomials, reduction, statistical query lower bounds, AMP, statistical physics, etc. \nI know the information-theoretic aspect and the info-comp gap is not the focus of this paper. \nBut when it comes to such a prototypical problem of tensor PCA, it's always good to say a few words about it so that the present results are properly positioned in the literature.\n\nQUESTIONS:\n1. In equation (1.3), should (r) in the superscript be (i)?\n\n2. Line 58 'Where we estimate v_j^{(1)} by the first right singular vector...'. Here by 'first', I suppose lambda1 >= lambda2 >= ... is assumed. Am I right?\n\n3. The set {v_j(k)} is assumed to be orthonormal. How strong is this assumption? It seems a bit more realistic to assume v_j^{(k)}'s are in general position. Also, my impression is that this will create quite some technical challenges. Correct me if I'm wrong. \n\n3. Some theorem/lemma statements are not sufficiently clear. E.g., Theorem 2.1, line 120, 122, does 'recovered' still refer to partial recovery as in line 118? I suggest state it more formally, at least in the theorem statement. Please check thoroughly for similar issues. \n\n4. Lemma 3.2 is the standard Gaussian conditioning lemma. It'll be nice to put a reference there. The standard one might be Lemma 11 of Bayati--Montanari (https://arxiv.org/abs/1001.3448), though there might be better ones. Please check the literature. \n\n5. Proof of Theorem 3.1, I feel it's actually better to do the proof for general k instead of k=3, maybe in the full version of the paper if the authors wish. \n\n6. The successive contraction algorithm is indeed quite natural. It reminds me of something called decimation AMP studied here https://arxiv.org/abs/2306.01412. That's in the context of *high*-rank matrix estimation (using AMP, but never mind that) and it estimates each component of the planted matrix sequentially and subtract it out for the next one. \n\n7. I'm not sure if the authors are aware of the following line of work that analyzes MLE (which is computationally expensive) for tensor PCA using a novel technique. The results are not completely new since MLE has been well-understood and moreover their results require the assumption that the limit of overlap exists. However, their techniques involves standard RMT and seem conceptually more accessible. The authors may want to look into or even consider citing some of them. \n- https://arxiv.org/abs/2108.00774 (square case)\n- https://arxiv.org/abs/2112.12348 (rectangular case)\n- and related works by roughly the same authors\n- https://arxiv.org/abs/2110.01652 (spectra of contracted tensors)\n\n\nMinor things:\n- I found it slightly weird that the citations are in the format (1), (2), etc. Note that the same format is also used for bullet points (see line 95, 110). Very similar notation is also used for equation numbers (1.1), (1.2). To avoid confusion, I suggest use the more standard [1], [2] or [ABC20], [XYZ99] format for citation. \n\n- Sometimes the first letter of 'proof of Theorem XXX' is not capitalized.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

comments: SUMMARY:
This paper is concerned of the tensor PCA problem (low-rank tensor recovery with Gaussian noise), in which the authors proposed three new ways to approach the problem with theoretical guarantees.

STRENGTHS:
1. This paper uses a succinct way to introduce to readers 3 different approaches of tensor PCA, all with formal guarantees. Therefore, a reader can quickly identify the approach they personally need.

2. All of the results are based on well-established matrix PCA results, so it is easier for readers to understand, and also gives people stronger confidence to use, as tensor analysis is notoriously hard.

WEAKNESSES:
1. Although the results themselves are fine, and solid to my eyes, but none of them provide new insights to the problem, nor did the authors come up with new or innovative techniques. It just seems to me that all this paper is doing is adapting a few matrix PCA techniques to the tensor regime, and only perform minimal changes to the procedure so that the framework can be used. This does not mean the results themselves are inferior, but the merit this paper brings is very limited beyond the verbatim theorems.

2. The paper presented three different approaches, but did not compare/contrast the three, so unfamiliar readers may not know which is the best technique to apply.

3. The simulation is limited (however, this is mostly a theoretical paper, so acceptable).

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies tensor principal component analysis (PCA) in a high-dimensional asymptotic framework, where each array dimension tends to infinity. The authors analyze matricization-based approaches, which convert tensors to matrices and then apply spectral methods. They fully analyze tensor unfolding or reshaping and the partial trace approach, discovering their asymptotic equivalence in performance. They identify sharp thresholds in signal-to-noise ratio above which these algorithms partially recover the signal and provide exact formulas for their limiting performance. Finally, they introduce an iterative algorithm based on successive contractions of the tensor, which achieves exact signal recovery above the same thresholds characterizing partial recovery of unfolding and partial trace. The authors' approach relies on fundamental random matrix theory results.\n\nSTRENGTHS:\nThe main strengths of the paper are:\n1. The paper provides a comprehensive analysis of tensor PCA algorithms in a high-dimensional asymptotic framework, allowing for tensors of diverse dimensions and unrelated signals.\n2. The paper introduces an iterative algorithm based on successive contractions of the tensor, which achieves exact signal recovery above the best-known algorithmic threshold.\n3. The paper provides exact formulas for the limiting performance of the algorithms and identifies sharp thresholds in the signal-to-noise ratio above which these algorithms partially recover the signal.\n\nWEAKNESSES:\nThe main weaknesses of the paper are:\n1. The presented successive contraction is demonstrated to achieve exact recovery only asymptotically, while a non-asymptotic result would be more interesting to investigate. In fact, the authors should compare the obtained performance of the successive contraction method in a non-asymptotic regime with the MLE which was analyzed in [1].\n2. It is unclear how the successive contraction method differs from the tensor power iteration. In fact, [2] showed that the latter achieves also exact recovery above the algorithmic threshold when initialized with signal estimates using the tensor unfolding method.\n\n[1] Mohamed El Amine Seddik, Maxime Guillaud, and Romain Couillet. “When Random Tensors meet Random Matrices”. In: arXiv preprint arXiv:2112.12348 (2021)\n\n[2] Arnab Auddy and Ming Yuan. “On Estimating Rank-One Spiked Tensors in the Presence of Heavy Tailed Errors”. In: arXiv preprint arXiv:2107.09660 (2021)\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nThe main limitations of the paper are:\n1. The paper considers an asymptotic analysis for Tensor PCA assuming the dimension of the tensor to grow to infinity, while a non-asymptotic analysis or rather a quantification of the fluctuations would be interesting to investigate.\n2. Most of the presented results in the paper are relatively known in the literature (Tensor Unfolding and Partial Trace).\n3. Lack of comparison of the presented successive contraction method with the tensor power iteration algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}