Title: Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods
Venue: NeurIPS 2024
Accepted: yes
Keywords: Matrix denoising

heteroscedasticity

spectral methods

approximate message passing

statistical physics
Abstract: We study the matrix denoising problem of estimating the singular vectors of a rank-$1$ signal corrupted by noise with both column and row correlations. Existing works are either unable to pinpoint the exact asymptotic estimation error or, when they do so, the resulting approaches (e.g., based on whitening or singular value shrinkage) remain vastly suboptimal. On top of this, most of the literature has focused on the special case of estimating the left singular vector of the signal when the noise only possesses row correlation (one-sided heteroscedasticity). In contrast, our work establishes the information-theoretic and algorithmic limits of matrix denoising with doubly heteroscedastic noise. We characterize the exact asymptotic minimum mean square error, and design a novel spectral estimator with rigorous optimality guarantees: under a technical condition, it attains positive correlation with the signals whenever information-theoretically possible and, for one-sided heteroscedasticity, it also achieves the Bayes-optimal error. Numerical experiments demonstrate the significant advantage of our theoretically principled method with the state of the art. The proofs draw connections with statistical physics and approximate message passing, departing drastically from standard random matrix theory techniques.

Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods

Abstract

We study the matrix denoising problem of estimating the singular vectors of a rank1 signal corrupted by noise with both column and row correlations. Existing works are either unable to pinpoint the exact asymptotic estimation error or, when they do so, the resulting approaches (e.g., based on whitening or singular value shrinkage) remain vastly suboptimal. On top of this, most of the literature has focused on the special case of estimating the left singular vector of the signal when the noise only possesses row correlation (one-sided heteroscedasticity). In contrast, our work establishes the information-theoretic and algorithmic limits of matrix denoising with doubly heteroscedastic noise. We characterize the exact asymptotic minimum mean square error, and design a novel spectral estimator with rigorous optimality guarantees: under a technical condition, it attains positive correlation with the signals whenever information-theoretically possible and, for one-sided heteroscedasticity, it also achieves the Bayes-optimal error. Numerical experiments demonstrate the significant advantage of our theoretically principled method with the state of the art. The proofs draw connections with statistical physics and approximate message passing, departing drastically from standard random matrix theory techniques.

1 Introduction

Matrix denoising is a central primitive in statistics and machine learning, and the problem is to recover a signal X ∈ R n × d from an observation A = X + W corrupted by additive noise W . This finds applications across multiple domains of sciences, e.g., imaging [24, 63], biology [14, 46] and astronomy [70, 5]. When X has low rank and W i.i.d. entries, A is the standard model for principal component analysis, typically referred to as the Johnstone spiked covariance model [42]. When n, d are both large and proportional, which corresponds to the most sample-efficient regime, its Bayes-optimal limits are well understood [52], and it has been established how to achieve them efficiently [56]. Minimax/non-asymptotic guarantees are also available in special cases, such as sparse PCA [18], Gaussian mixtures [72] and certain joint scalings of ( n, d ) [57].

However, in most applications, noise is highly structured and correlated, thereby calling for more realistic assumptions on W than having i.i.d. entries. A recent line of work addresses this concern by studying matrix denoising with heteroscedastic noise [1, 69, 33, 44, 26], resting on two basic ideas: whitening and singular value shrinkage. Whitening refers to multiplying the data matrix by the square root of the inverse covariance, in order to reduce the model to one with i.i.d. noise; and singular value shrinkage retains the singular vectors of the data while deflating the singular values to correct for the noise. Though the exact asymptotic performance of these algorithms has been derived [69, 33, 44, 26], their optimality is yet to be determined from a Bayesian standpoint. In fact, we will prove that whitening and shrinkage are not the correct way to approach Bayes optimality.

Main contributions. We focus on the prototypical model A = X + W , where X = λ n u ∗ v ∗⊤ is a rank1 signal, λ is the signal-to-noise ratio (SNR), and W = Ξ 1 / 2 ˜ W Σ 1 / 2 is doubly heterogeneous noise. Here u ∗ , v ∗ follow i.i.d. priors; ˜ W contains i.i.d. Gaussian entries; the covariance matrices Ξ , Σ capture column and row correlations; and we consider the typical high-dimensional regime in which n, d are both large and proportional. Our main results are summarized below.

We design an efficient spectral estimator to recover u ∗ , v ∗ , and we provide a precise asymptotic analysis of its performance, see Theorem 5.1. This estimator is given by the top singular vectors of a matrix obtained by carefully pre-processing A , see (5.3).

When the priors of u ∗ , v ∗ are standard Gaussian, we show in Corollary 5.2 that the spectral estimator above is optimal in the following sense: (i) under a technical condition, it achieves the optimal weak recovery threshold , namely its mean square error is non-trivial as soon as this is information-theoretically possible; (ii) it achieves the Bayes-optimal error for u ∗ (resp. v ∗ ) when Ξ (resp. Σ ) is the identity. These optimality guarantees follow from rigorously obtaining the asymptotic minimum mean square error (MMSE) for the estimation of the whitened signals Ξ -1 / 2 u ∗ and Σ -1 / 2 v ∗ , see Theorem 4.2.

Our spectral estimator only involves matrix multiplication and computing principal singular vectors. Practically, this can be efficiently done using standard SVD algorithms or power iteration [48]. For both one-sided and double heteroscedasticity, numerical experiments in Figures 2 and 3 show significant advantage of our spectral estimator for moderate SNRs over HeteroPCA [76] and shrinkagebased methods, i.e., Whiten-Shrink-reColor [44, 45], OptShrink [59], and ScreeNOT [27].

Proof techniques. We take a completely different route from classical approaches in statistics and random matrix theory (e.g., whitening and shrinkage), and instead exploit tools from statistical physics and the theory of approximate message passing. In particular, the MMSE for the whitened signals Ξ -1 / 2 u ∗ , Σ -1 / 2 v ∗ is obtained via an interpolation argument [10, 52, 53]. This result allows us to derive the weak recovery threshold for estimating the true signals u ∗ , v ∗ . Moreover, for one-sided heteroscedasticity, this MMSE coincides with that for estimating the true signal on the homoscedastic side. Evaluating the Bayes-optimal estimators requires solving high-dimensional integrals that are computationally intractable. To circumvent this issue, we propose an efficient spectral method that still enjoys optimality guarantees. Its design and analysis draw connections with a family of iterative algorithms called approximate message passing (AMP) [11, 32]. All our results are mathematically rigorous, with the only technical condition being '(5.1) implies σ ∗ 2 < 1 ' in Theorem 5.1 that we only managed to verify numerically, but not analytically; see Remark 5.1.

2 Related work

Research on matrix denoising in the homoscedastic case ( Ξ = I n , Σ = I d ) has a rich history, and in random matrix theory properties of the spectrum and eigenspaces of A have been studied exhaustively. Most prominently, the BBP phase transition phenomenon [4] (and its finite-sample counterpart [60]) unveils a threshold of the SNR λ above which a pair of outlier singular value and singular vector emerge. Under i.i.d. priors, the asymptotic Bayes-optimal estimation error has been derived [52, 53], rigorously justifying predictions from statistical physics [47]. The proof uses the interpolation method due to Guerra [35], originally developed in the context of mean-field spin glasses. Besides low-rank matrix estimation, this method (including its adaptive variant [10] and the Aizenman-Sims-Starr scheme [2]) has also been applied to a range of problems, including spiked tensor estimation [49], generalized linear models [9], stochastic block models [74] and group synchronization [73].

Moving to the heteroscedastic case, an active line of work concerns optimal singular value shrinkage methods [44, 33, 45, 69, 59, 26]. These methods can be regarded as a special family of rotationally invariant estimators, which apply a univariate function η : R ≥ 0 → R to each empirical singular value. An example widely employed by practitioners is the thresholding function η θ ( y ) = y 1 { y > θ } [27]. In the presence of noise heteroscedasticity, most of these results are based on whitening [43]. Another model of noise heterogeneity common in the literature takes W = ˜ W ◦ ∆ ◦ 1 / 2 , where ˜ W has i.i.d. Gaussian entries, ∆ is a deterministic block matrix with fixed (i.e., constant with respect to n, d ) number of blocks, and ◦ denotes the element-wise product. This means that the entries of the noise are independent but non-identically distributed, and they follow the variance profile ∆ . The corresponding low-rank perturbation A , known as a spiked inhomogeneous matrix, has attracted attention from both the information-theoretic [12, 66, 36] and the algorithmic sides [38, 50, 61].

Spiked inhomogeneous matrices have some connections with the model considered in this paper: if ∆ has rank 1 , such A can be realized by taking Ξ , Σ to be diagonal with suitable block structures. Finally, non-asymptotic results for the heteroscedastic and the inhomogeneous models have been derived in varying generality in [76, 82, 23, 1, 17]. We highlight that our paper is the first to establish information-theoretic and algorithmic limits for doubly heteroscedastic noise.

Our characterization of the spectral estimator relies on an AMP algorithm that converges to it by performing power iteration. AMP refers to a family of iterative procedures, whose performance in the high-dimensional limit is precisely characterized by a low-dimensional deterministic recursion called state evolution [11, 15]. Originally introduced for compressed sensing [28], AMP algorithms have been developed for various settings, including low-rank estimation [56, 31, 6] and inference in generalized linear models [64, 65, 71]. Beyond statistical estimation, AMP proves its versatility as both an efficient algorithm and a proof technique for studying e.g. posterior sampling [58], spectral universality [30], first order methods with random data [22], mismatched estimation [8], spectral estimators for generalized linear models [79, 80] and their combination with linear estimators [54].

3 Problem setup

Consider the following rank1 rectangular matrix estimation problem with doubly heteroscedastic noise where we observe

and aim to estimate u ∗ , v ∗ . The following assumptions are imposed throughout the paper. The dimensions n, d →∞ obey the proportional scaling n/d → δ ∈ (0 , ∞ ) , where δ is the aspect ratio. The SNR λ ∈ [0 , ∞ ) is a known constant (relative to n, d ). The signals ( u ∗ , v ∗ ) ∼ P ⊗ n ⊗ Q ⊗ d have i.i.d. priors, where P, Q are distributions on R with mean 0 and variance 1 . The unknown noise matrix has the form W = Ξ 1 / 2 ˜ W Σ 1 / 2 ∈ R n × d , with ˜ W i,j i . i . d . ∼ N (0 , 1 /n ) independent of ( u ∗ , v ∗ ) . The covariances Ξ ∈ R n × n , Σ ∈ R d × d are known, deterministic, 1 strictly positive definite and satisfy

Their empirical spectral distributions (ESD) converge (as n, d →∞ s.t. n/d →∞ ) weakly to the laws of the random variables Ξ and Σ . Furthermore, ∥ Ξ ∥ 2 , ∥ Σ ∥ 2 are uniformly bounded over d . The supports of Ξ , Σ are compact subsets of (0 , ∞ ) . For all ε > 0 , there exists d 0 ∈ N s.t. for all d ≥ d 0 ,

The trace assumption (3.2) on the covariances is for normalization purposes since the values of the traces, if not 1 , can be absorbed into λ . The support assumption (3.3) excludes outliers in the spectra of covariances which may contribute to undesirable spikes in A [69].

4 Information-theoretic limits

For mathematical convenience, in this section, we switch to an equivalent rescaled model

where γ := λ 2 and Z = √ n ˜ W contains i.i.d. elements Z i,j i . i . d . ∼ N (0 , 1) . Abusing terminology, we refer to γ as the SNR of Y . Define also α := 1 /δ ∈ (0 , ∞ ) so that d/n → α . The scaling of the parameters in (4.1) turns out to be more convenient for presenting the results in this section. Results for Y can be easily translated to A by a change of variables.

Let ˜ u ∗ := Ξ -1 / 2 u ∗ and ˜ v ∗ := Σ -1 / 2 v ∗ denote the whitened signals. The main result of this section is Theorem 4.2, which characterizes the performance of the matrix minimum mean square error

(MMSE) associated to the estimation of ˜ u ∗ ( ˜ v ∗ ) ⊤ , ˜ u ∗ ( ˜ u ∗ ) ⊤ and ˜ v ∗ ( ˜ v ∗ ) ⊤ , via the corresponding Bayes-optimal estimators:

Our characterization involves a pair of parameters ( q ∗ u , q ∗ v ) ∈ R 2 ≥ 0 defined as the largest solution to

Here and throughout the paper, all expectations involving Ξ , Σ are computed as integrals against the limiting spectral distributions of Ξ , Σ .

The proposition below, proved in Appendix A, justifies the existence of the solution to (4.5) and identifies when a non-trivial solution emerges.

Proposition 4.1. The fixed point equation (4.5) always has a trivial solution (0 , 0) . There exists a non-trivial solution ( q ∗ u , q ∗ v ) ∈ R 2 > 0 if and only if

in which case the non-trivial solution is unique.

We are now ready to state our main result on the MMSE.

Theorem 4.2. Assume P = Q = N (0 , 1) . For almost every γ > 0 ,

We note that

where the last step follows from Proposition G.2. This quantity represents the trivial error in the estimation of ˜ u ∗ ( ˜ v ∗ ) ⊤ , which is achieved by the all-0 estimator. Analogous considerations hold for ˜ u ∗ ( ˜ u ∗ ) ⊤ and ˜ v ∗ ( ˜ v ∗ ) ⊤ , for which the trivial estimation error is E [ Ξ -1 ] 2 and E [ Σ -1 ] 2 , respectively. Thus, Proposition 4.1 and Theorem 4.2 identify (4.6) as the condition for non-trivial estimation, and the smallest γ that satisfies (4.6) gives the weak recovery threshold .

We show below that the weak recovery threshold is the same for the estimation of the true signals u ∗ v ∗⊤ , u ∗ u ∗⊤ and v ∗ v ∗⊤ . In this case, since the signal priors are Gaussian, using the same passages as in (4.9) one has that the trivial estimation error for u ∗ v ∗⊤ , u ∗ u ∗⊤ and v ∗ v ∗⊤ is always equal to 1 .

Corollary 4.3. Assume P = Q = N (0 , 1) . The MMSE associated to the estimation of u ∗ v ∗⊤ is non-trivial, i.e,

if and only if (4.6) holds. The same result holds for the MMSE of u ∗ u ∗⊤ and v ∗ v ∗⊤ .

Proof strategy. To derive the characterizations in Theorem 4.2, we write the posterior distribution of u ∗ , v ∗ given Y in a Gibbs form, i.e., its density is the exponential of a Hamiltonian normalized by a partition function. The interpolation argument relates the log-partition function (also referred to as the 'free energy') of the posterior to that of the posteriors of two Gaussian location models. Since i.i.d. Gaussianity is key to this approach, the challenge is to handle noise covariances. Our idea is

to incorporate the covariances into the priors. In terms of the Hamiltonian, the model is equivalent to the estimation of the whitened signals Ξ -1 / 2 u ∗ , Σ -1 / 2 v ∗ , whose priors have covariances, in the presence of i.i.d. Gaussian noise. We then manage to carry out the interpolation argument for the equivalent model and evaluate the free energy of the corresponding Gaussian location models.

Specifically, let us starting by writing down the expression of the posterior distribution after setting up some notation. For u ∈ R n , v ∈ R d , let ˜ u := Ξ -1 / 2 u, ˜ v := Σ -1 / 2 v . Define the densities

where the determinant factors ensure that the integrals equal 1 . With P = Q = N (0 , 1) , we have ˜ P = N (0 n , Ξ -1 ) , ˜ Q = N (0 d , Σ -1 ) , and from Bayes' rule the posterior of ( u ∗ , v ∗ ) given Y is

where the Hamiltonian and the partition function are given respectively by

Define the free energy as

The major technical step is to characterize F n ( γ ) in the large n limit in terms of a bivariate functional F introduced below. This is the core component to derive the MMSE characterization.

For a positive random variable Σ subject to the conditions in Section 3, let

As shown in Appendix B, ψ Σ ( γ ) is the limiting free energy of a Gaussian channel, in which one wishes to estimate x ∗ ∈ R n from the observation Y = √ γx ∗ +Σ 1 / 2 Z corrupted by anisotropic Gaussian noise with covariance Σ . Using (4.15), let us define the replica symmetric potential F :

and the set of critical points of F :

where the last equality is a direct calculation of ψ ′ Ξ , ψ ′ Σ . The following result, proved in Appendix C, shows that the limit of F n ( γ ) is given by a dimension-free variational problem involving F ( q u , q v ) .

Theorem 4.4 (Free energy) . Assume P = Q = N (0 , 1) . Then, we have

and sup q v inf q u and sup ( q u ,q v ) are achieved by the same ( q ∗ u , q ∗ v ) in Proposition 4.1.

Remark 4.1 (Equivalent models) . Informally, the above result says that the matrix model (4.1) is equivalent at the level of Hamiltonian to the following two statistically uncorrelated vector models:

with q ∗ u , q ∗ v the largest solution to (4.5) and ( u ∗ , v ∗ , Z u , Z v ) ∼ P ⊗ n ⊗ Q ⊗ d ⊗N (0 n , I n ) ⊗N (0 d , I d ) .

Remark 4.2 (Gaussian priors) . Theorem 4.4 crucially relies on having Gaussian priors P, Q . This assumption is mainly used to derive single-letter (i.e., dimension-free) expressions of the free energy of the vector models in (4.17) which, under Gaussian priors, are nothing but Gaussian integrals. The free energy, and hence the MMSE, are expected to be sensitive to the priors. Indeed, this is already the case in the homoscedastic setting Ξ = I n , Σ = I d [52]. An extension towards general i.i.d. priors is a challenging open problem and, in fact, without posing additional assumptions on Ξ , Σ , it is unclear whether a single-letter expression for free energy and MMSE is possible.

At this point, the MMSE can be derived from the above characterization of free energy. Indeed, let

The envelope theorem [51, Corollary 4] ensures that D ( α ) is equal to R > 0 up to a countable set. Using algebraic relations between free energy and MMSE, we prove (4.7) and (4.8) for all γ ∈ D ( α ) (and, thus, for almost every γ > 0 ). Then, using the Nishimori identity (Proposition G.4) and the fact that the ESDs of Ξ , Σ are upper and lower bounded by constants independent of n and d , Corollary 4.3 also follows. The formal arguments are contained in Appendix D.

5 Spectral estimator

This section introduces a spectral estimator that meets the weak recovery threshold and, for one-sided heteroscedasticity, attains the Bayes-optimal error. Suppose that the following condition holds

which is equivalent to (4.6). Under this condition, the fixed point equations (4.5) have a unique pair of positive solutions ( q ∗ u , q ∗ v ) . For convenience, we also define the rescalings µ ∗ := λq ∗ v /δ, ν ∗ := λq ∗ u , and the auxiliary quantities

Now, we pre-process the data matrix A as

from which we obtain the spectral estimators

where u 1 ( · ) /v 1 ( · ) denote the top left/right singular vectors and

Note that η u , η v > 0 , provided that (5.1) holds. The pre-processing of A in (5.3) and the form of the spectral estimators in (5.4) come from the derivation of a suitable AMP algorithm, and they are discussed at the end of the section. We finally defer to Appendix E.3 the definition of the scalar quantity σ ∗ 2 obtained via a fixed point equation depending only on Ξ , Σ , λ, δ , see (E.26) for details.

Our main result, Theorem 5.1, shows that, under the criticality condition (5.1), the matrix A ∗ exhibits a spectral gap between the top two singular values, and it characterizes the performance of the spectral estimators in (5.4), proving that they achieve weak recovery of u ∗ and v ∗ , respectively.

Theorem 5.1. Suppose that (5.1) holds and that, for any c > 0 ,

Figure 1: Top two singular values of A ∗ in (5.3), where d = 4000 , δ = 4 and each simulation is averaged over 10 i.i.d. trials. The singular values computed experimentally ('sim' in the legends and × in the plots) closely match our theoretical prediction in (5.7) ('thy' in the legends and solid curves with the same color in the plots). The threshold λ ∗ is such that equality holds in (5.1). We note that the green curve corresponding to σ ∗ 2 is smaller than 1 for λ > λ ∗ , i.e., when (5.1) holds.

where Ξ ∗ := λ λ ( µ ∗ + b ∗ )+Ξ , Σ ∗ := λ λ ( ν ∗ + c ∗ )+Σ and s := c · sup supp(Σ ∗ ) . Let A ∗ , ̂ u, ̂ v, σ ∗ 2 be defined in (5.3) , (5.4) and (E.26) , and σ i ( A ∗ ) denote the i -th largest singular value of A ∗ . Then, if σ ∗ 2 < 1 , the following limits hold in probability:

Remark 5.1 (Assumptions) . To guarantee a spectral gap for A ∗ and the weak recoverability of u ∗ , v ∗ via the proposed spectral method, we also require the algebraic condition σ ∗ 2 < 1 . We conjecture that this condition is implied by (5.1), and we have verified that this is the case in all our numerical experiments (see Figure 1 for two concrete examples). The additional assumption (5.6) is a mild regularity condition on the covariances. It ensures that the densities of Ξ ∗ , Σ ∗ decay sufficiently slowly at the edges of the support, so that σ ∗ 2 is well-posed [79].

Remark 5.2 (Signal priors) . Theorem 5.1 does not require the prior distributions P, Q to be Gaussian, and it is valid for any i.i.d. prior with mean 0 and variance 1 .

On the one hand, Corollary 4.3 shows that, if (5.1) is violated, the problem is information-theoretically impossible, i.e., no estimator achieves non-trivial error. On the other hand, Theorem 5.1 exhibits a pair of estimators that achieves non-trivial error as soon as (5.1) holds - under the additional assumption σ ∗ 2 < 1 which we conjecture to be implied by (5.1). Thus, the spectral method in (5.4) is optimal in terms of weak recovery threshold. Though such estimators do not attain the optimal error, when both priors are Gaussian and Ξ = I n , ̂ u ̂ u ⊤ is the Bayes-optimal estimate for u ∗ u ∗⊤ .

Corollary 5.2. Assume P = Q = N (0 , 1) , and consider the setting of Theorem 5.1 with the additional assumption Ξ = I n . Then, η u = √ q ∗ u , i.e., ̂ u ̂ u ⊤ achieves the MMSE for u ∗ u ∗⊤ .

The claim readily follows by noting that, when Ξ = I n , the first equation in (4.5) becomes

where the last equality is by the definition (5.5) of η u . Let us highlight that, even if Ξ = I n , ̂ u still makes non-trivial use of the other covariance Σ 1 / 2 . At the information-theoretic level, this is reflected by the fact that Σ 1 / 2 enters q ∗ u through the fixed point equations (4.5). Therefore, even though the matrix model in (4.1) is equivalent to a pair of uncorrelated vector models in (4.17) in the sense of the free energy, the tasks of estimating u ∗ and v ∗ cannot be decoupled.

(a) Normalized correlation with

u

∗

(b) Normalized correlation with

v

∗

(c) Matrix MSE for

u

∗

v

∗⊤

(a) Normalized correlation with

u

∗

(b) Normalized correlation with

v

∗

(c) Matrix MSE for

u

∗

v

∗⊤

Numerical experiments. Figures 2 and 3 demonstrate the advantage of our method over existing approaches, and they display an accurate agreement between simulations ('sim' in the legends and × in the plots) and the theoretical predictions of Theorem 5.1 ('thy' in the legends and solid curves with the same color in the plots), both plotted as a function of λ . In both figures, n = 4000 , d = 2000 (so δ = 2 ), and P = Q = N (0 , 1) . Each data point is computed from 20 i.i.d. trials and error bars are reported at 1 standard deviation. We let Ξ be either the identity or a Toeplitz matrix [77, 41, 19], i.e., Ξ i,j = ρ | i -j | with ρ = 0 . 9 . We let Σ be a circulant matrix [40, 39]: the first row has 1 in the first position, c = 0 . 0078 in the second through ( ℓ +1) -st position and in the last ℓ positions ( ℓ = 300 ), with the remaining entries being 0 ; for 2 ≤ i ≤ d , the i -th row is a cyclic shift of the ( i -1) -st row to the right by 1 position. Both matrices satisfy (5.6) and the conditions of Section 3.

Our spectral estimator outperforms all other approaches: Leeb-Romanov [44], OptShrink [59], ScreeNOT [27], and HeteroPCA [76] in the one-sided heteroscedastic case (Figure 2); Leeb [45], OptShrink, and ScreeNOT in the doubly heteroscedastic case (Figure 3). When computing the normalized correlation with the signals (left/right overlap), the performance of Leeb-Romanov and Leeb is the same as the estimators Ξ 1 / 2 u 1 (Ξ -1 / 2 A Σ -1 / 2 ) , Σ 1 / 2 v 1 (Ξ -1 / 2 A Σ -1 / 2 ) referred to as 'whiten' in Figures 2a and 2b; the performance of OptShrink and ScreeNOT is the same as the estimators u 1 ( A ) , v 1 ( A ) referred to as 'vanilla' in Figures 3a and 3b. The advantage of our approach (in black) is especially significant at low SNR; as SNR increases, Leeb-Romanov and Leeb (in red) achieve similar performance; a much larger SNR ( > 2 and > 3 in Figures 2 and 3) is required by HeteroPCA, OptShrink and ScreeNOT (in magenta, blue and green) to perform comparably.

Finally, Figure 4 shows the presence of spectral outliers in A ∗ and their absence in A at a fixed λ .

Proof strategy. The design and analysis of the spectral estimator in (5.4) comprise two steps, detailed in Appendix E. The first step is to present an AMP algorithm dubbed Bayes-AMP for matrix denoising with doubly heteroscedastic noise. Specifically, its iterates are updated as

(a)

λ

= 1

,

Ξ =

I

n

and

Σ

a Toeplitz matrix with

ρ

= 0

.

9

.

where ∇ denotes the Jacobian matrix, and the functions g ∗ t , f ∗ t +1 are specified below in (5.12). As common in AMP algorithms, the iterates (5.11) are accompanied with a state evolution which accurately tracks their behavior via a simple deterministic recursion: the joint empirical distribution of ( u ∗ , v ∗ , u t , v t +1 ) converges to the random variables ( U ∗ , V ∗ , U t , V t +1 ) , see Proposition E.1 for a formal statement and the recursive description of the laws of such random variables. Then, the name 'Bayes-AMP' is motivated by the fact that g ∗ t , f ∗ t +1 are the posterior-mean denoisers given by

Remarkably, Bayes-AMP operates on Ξ -1 A Σ -1 , as opposed to the widely adopted ansatz of considering the whitened matrix Ξ -1 / 2 A Σ -1 / 2 . The advantage of operating on Ξ -1 A Σ -1 is that the fixed point of the corresponding state evolution matches the extremizers of the free energy in (4.5). This would not be the case if Bayes-AMP used the whitening Ξ -1 / 2 A Σ -1 / 2 . Indeed, one can repeat the analysis of an AMP that operates on Ξ -1 / 2 A Σ -1 / 2 . The fixed point equations of the resulting state evolution do not match the information-theoretically optimal one in (4.5). In particular, the weak recovery threshold coming out of this approach is strictly larger than the optimal one in (4.6), as long as at least one of Ξ , Σ is not a multiple of the identity. Since these derivations led to suboptimal results, the details were left out from the paper.

The design of Bayes-AMP and the proof of its state evolution follow a two-step reduction detailed in Appendix F. Using a change of variables, we show in Appendix F.2 that Bayes-AMP can be realized by an auxiliary AMP with non-separable denoising functions (meaning that g t , f t +1 cannot be written as univariate functions applied component-wise) operating on Ξ -1 / 2 A Σ -1 / 2 = λ n ˜ u ∗ ( ˜ v ∗ ) ⊤ + ˜ W . Then, in Appendix F.1 we simulate the auxiliary AMP using a standard AMP operating on the i.i.d. Gaussian matrix ˜ W , whose state evolution has been established in [13, 34].

However, Bayes-AMP by itself is not a practical algorithm since it needs a warm start, i.e., an initialization that achieves non-trivial error. Thus, the second step is to design a spectral estimator that solves the fixed point equation of Bayes-AMP, which turns out to be an eigen-equation for A ∗ .

To offer the readers an intuition on how the spectral estimators arise from Bayes-AMP, we now heuristically derive the form (5.3) of A ∗ and the expression (5.4) of the spectral estimator. To do so, we note that the largen limits of c t , b t +1 coincide with the auxiliary quantities c ∗ , b ∗ defined in (5.2).

Furthermore, when the priors of u ∗ , v ∗ are Gaussian, (5.12) reduces to

where we recall that µ ∗ = λq ∗ v /δ and ν ∗ = λq ∗ u are rescalings of the non-trivial solution ( q ∗ u , q ∗ v ) of (4.5). Denoting by u, v the fixed points of the iteration (5.11), after some manipulations we have g (Ξ) u = A ∗ f (Σ) v, f (Σ) v = A ∗⊤ g (Ξ) u, where A ∗ is given in (5.3) and

This suggests that A ∗ has top singular value equal to 1 and ( g (Ξ) u, f (Σ) v ) are aligned with the corresponding singular vectors ( u 1 ( A ∗ ) , v 1 ( A ∗ )) . Moreover, state evolution implies that the distribution of the fixed point ( u, v ) is close to that of

with ( w u , w v ) ∼ N (0 n , Ξ -1 ) ⊗N (0 d , Σ -1 ) independent of u ∗ , v ∗ . Thus, to obtain estimates of ( u ∗ , v ∗ ) , we take (Ξ g (Ξ) -1 u 1 ( A ∗ ) , Σ f (Σ) -1 v 1 ( A ∗ )) and suitably rescale their norm, which leads to the expressions in (5.4). More details on the above heuristics are discussed in Appendix E.2.

The most outstanding step remains to make the heuristics rigorous. This involves proving that Ξ u t , Σ v t +1 are aligned with the proposed spectral estimator, which allows for a performance characterization via state evolution. The formal argument is carried out in Appendix E.4.

6 Concluding remarks

In this work, we establish information-theoretic limits and propose an efficient spectral method with optimality guarantees, for matrix estimation with doubly heteroscedastic noise. On the one hand, under Gaussian priors, we give a rigorous characterization of the MMSE; on the other hand, we present a spectral estimator that (i) achieves the information-theoretic weak recovery threshold, and (ii) is Bayes-optimal for the estimation of one of the signals, when the noise is heteroscedastic only on the other side. While our analysis focuses on rank-1 estimation, we expect that all results admit proper extensions to rankr signals, where r is a constant independent on n, d .

The design and analysis of the spectral estimator draws connections with approximate message passing and, along the way, we introduce a Bayes-AMP algorithm which could be of independent interest. In this paper, we employ Bayes-AMP solely as a proof technique. However, one could use the spectral method designed here as an initialization of Bayes-AMP itself, after suitably correcting its iterates. This strategy has been successfully carried out for i.i.d. Gaussian noise in [56] and for rotationally invariant noise in [55, 81]. Bayes-AMP is well equipped to exploit signal priors more informative than the Gaussian one, and AMP algorithms are known to achieve the informationtheoretically optimal estimation error for low-rank matrix inference [56, 6]. Nevertheless, we point out two obstacles towards doing so in the presence of doubly heteroscedastic noise. First, for general priors, establishing the information-theoretic limits remains a challenging open problem, and it is unclear whether a low-dimensional characterization of the free energy (and, hence, of the MMSE) is possible. Second, even for Gaussian priors, Bayes-AMP reduces to the proposed spectral estimator, which is not Bayes-optimal for the general case of doubly heteroscedastic noise.

Finally, the proposed spectral estimator makes non-trivial use of the covariances Ξ , Σ , which are assumed to be known. When such matrices possess additional structure - e.g., they are sparse [21], their inverses are sparse [19] or they are circulant or Toeplitz [75] - their consistent estimation is possible, see also the survey [20]. However, in general, Ξ , Σ cannot be consistently estimated from the data when n and d grow proportionally. Thus, a challenging open problem is to construct estimators that retain comparable performance without knowing the noise covariances. The paper [33] addresses the challenge of unknown covariances by considering a modified model where one additionally observes an independent copy of noise. The statistician can then estimate the covariance from the noise-only observation and use it as a surrogate of the true covariance for estimating the signals from the spiked model. It is possible to derive similar results in the doubly heteroskedastic setting considered in our paper. If the covariances are completely unknown, then our model (with Gaussian priors) is equivalent to a spiked matrix model with a certain bi-rotationally invariant noise. This problem is expected to exhibit rather different behaviors than when covariances are known, see [7, 29] for recent progress on understanding the statistical and computational limits for such models.

Acknowledgments and Disclosure of Funding

YZ thanks Shashank Vatedka for discussions at the early stage of this project. MM thanks Jean Barbier for sharing his insights into the interpolation argument. This research is partially supported by the 2019 Lopez-Loreta Prize and by the Interdisciplinary Projects Committee (IPC) at the Institute of Science and Technology Austria (ISTA). This work was done in part while the authors were visiting the Simons Institute for the Theory of Computing.

References

Joshua Agterberg, Zachary Lubberts, and Carey E. Priebe. Entrywise estimation of singular vectors of low-rank matrices with heteroskedasticity and dependence. IEEE Trans. Inform. Theory , 68(7):4618-4650, 2022. 1, 3

Michael Aizenman, Robert Sims, and Shannon L. Starr. Extended variational principle for the sherrington-kirkpatrick spin-glass model. Phys. Rev. B , 68:214403, Dec 2003. 2

Z. D. Bai and Y. Q. Yin. Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix. Ann. Probab. , 21(3):1275-1294, 1993. 42, 48, 49

Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. , 33(5):1643-1697, 2005. 2

Stephen Bailey. Principal component analysis with noisy and/or missing data. Publications of the Astronomical Society of the Pacific , 124(919):1015, sep 2012. 1

Jean Barbier, Francesco Camilli, Marco Mondelli, and Manuel Sáenz. Fundamental limits in structured principal component analysis and how to reach them. Proc. Natl. Acad. Sci. USA , 120(30):Paper No. e2302028120, 7, 2023. 3, 10

Jean Barbier, Francesco Camilli, Marco Mondelli, and Yizhou Xu. Information limits and thouless-anderson-palmer equations for spiked matrix models with structured noise. CoRR , abs/2405.20993, 2024. 10

Jean Barbier, TianQi Hou, Marco Mondelli, and Manuel Saenz. The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation? In Advances in Neural Information Processing Systems , volume 35, pages 36733-36747, 2022. 3

Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proc. Natl. Acad. Sci. USA , 116(12):5451-5460, 2019. 2

Jean Barbier and Nicolas Macris. The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference. Probab. Theory Related Fields , 174(3-4):1133-1185, 2019. 2, 17

Mohsen Bayati and Andrea Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Trans. Inform. Theory , 57(2):764-785, 2011. 2, 3

Joshua K. Behne and Galen Reeves. Fundamental limits for rank-one matrix estimation with groupwise heteroskedasticity. In International Conference on Artificial Intelligence and Statistics , pages 8650-8672, 2022. 2

Raphaël Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approximate message passing with non-separable functions. Inf. Inference , 9(1):33-79, 2020. 9, 46

Tejal Bhamre, Teng Zhang, and Amit Singer. Denoising and covariance estimation of single particle cryo-em images. Journal of Structural Biology , 195(1):72-81, 2016. 1

Erwin Bolthausen. An iterative construction of solutions of the TAP equations for the Sherrington-Kirkpatrick model. Comm. Math. Phys. , 325(1):333-366, 2014. 3

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities . Oxford University Press, Oxford, 2013. 52

T. Tony Cai, Rungang Han, and Anru R. Zhang. On the non-asymptotic concentration of heteroskedastic Wishart-type matrix. Electron. J. Probab. , 27:Paper No. 29, 40, 2022. 3

T. Tony Cai, Zongming Ma, and Yihong Wu. Sparse PCA: optimal rates and adaptive estimation. Ann. Statist. , 41(6):3074-3110, 2013. 1

T. Tony Cai, Zhao Ren, and Harrison H. Zhou. Optimal rates of convergence for estimating Toeplitz covariance matrices. Probab. Theory Related Fields , 156(1-2):101-143, 2013. 8, 10

T. Tony Cai, Zhao Ren, and Harrison H. Zhou. Estimating structured high-dimensional covariance and precision matrices: optimal rates and adaptive estimation. Electron. J. Stat. , 10(1):1-59, 2016. 10

T. Tony Cai and Harrison H. Zhou. Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist. , 40(5):2389-2420, 2012. 10

Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data. arXiv preprint arXiv:2112.07572 , 2021. 3

Chen Cheng, Yuting Wei, and Yuxin Chen. Tackling small eigen-gaps: fine-grained eigenvector estimation and inference under heteroscedastic noise. IEEE Trans. Inform. Theory , 67(11):73807419, 2021. 3

Lucilio Cordero-Grande, Daan Christiaens, Jana Hutter, Anthony N. Price, and Jo V. Hajnal. Complex diffusion-weighted image estimation via matrix recovery under general noise models. NeuroImage , 200:391-404, 2019. 1

Romain Couillet and Walid Hachem. Analysis of the limiting spectral measure of large random matrices of the separable covariance type. Random Matrices Theory Appl. , 3(4):1450016, 23, 2014. 39

Xiucai Ding, Yun Li, and Fan Yang. Eigenvector distributions and optimal shrinkage estimators for large covariance and precision matrices. arXiv preprint arXiv:2404.14751 , 2024. 1, 2

David Donoho, Matan Gavish, and Elad Romanov. ScreeNOT : exact MSE-optimal singular value thresholding in correlated noise. Ann. Statist. , 51(1):122-148, 2023. 2, 8

David L. Donoho, Arian Maleki, and Andrea Montanari. Message passing algorithms for compressed sensing. Proceedings of the National Academy of Sciences , 106:18914-18919, 2009. 3

Rishabh Dudeja, Songbin Liu, and Junjie Ma. Optimality of approximate message passing algorithms for spiked matrix models with rotationally invariant noise. CoRR , abs/2405.18081, 2024. 10

Rishabh Dudeja, Subhabrata Sen, and Yue M Lu. Spectral universality of regularized linear regression with nearly deterministic sensing matrices. IEEE Transactions on Information Theory , 2024. 3

Zhou Fan. Approximate message passing algorithms for rotationally invariant matrices. The Annals of Statistics , 50(1):197-224, 2022. 3

Oliver Y Feng, Ramji Venkataramanan, Cynthia Rush, Richard J Samworth, et al. A unifying tutorial on approximate message passing. Foundations and Trends® in Machine Learning , 15(4):335-536, 2022. 2

Matan Gavish, William Leeb, and Elad Romanov. Matrix denoising with partial noise statistics: optimal singular value shrinkage of spiked F-matrices. Inf. Inference , 12(3):Paper No. iaad028, 46, 2023. 1, 2, 10

Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. Inf. Inference , 12(4):Paper No. iaad020, 67, 2023. 9, 46

Francesco Guerra. Broken replica symmetry bounds in the mean field spin glass model. Comm. Math. Phys. , 233(1):1-12, 2003. 2

Alice Guionnet, Justin Ko, Florent Krzakala, and Lenka Zdeborová. Low-rank matrix estimation with inhomogeneous noise. arXiv preprint arXiv:2208.05918 , 2022. 2

Philip Hartman. Ordinary differential equations , volume 38. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2002. 27

David Hong, Fan Yang, Jeffrey A. Fessler, and Laura Balzano. Optimally weighted PCA for high-dimensional heteroscedastic data. SIAM J. Math. Data Sci. , 5(1):222-250, 2023. 2

Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for highdimensional regression. J. Mach. Learn. Res. , 15:2869-2909, 2014. 8

Adel Javanmard and Andrea Montanari. Hypothesis testing in high-dimensional regression under the Gaussian random design model: asymptotic theory. IEEE Trans. Inform. Theory , 60(10):6522-6554, 2014. 8

Adel Javanmard and Andrea Montanari. Debiasing the Lasso: optimal sample size for Gaussian designs. Ann. Statist. , 46(6A):2593-2622, 2018. 8

Iain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. , 29(2):295-327, 2001. 1

Boris Landa, Thomas T. C. K. Zhang, and Yuval Kluger. Biwhitening reveals the rank of a count matrix. SIAM J. Math. Data Sci. , 4(4):1420-1446, 2022. 2

William Leeb and Elad Romanov. Optimal spectral shrinkage and PCA with heteroscedastic noise. IEEE Trans. Inform. Theory , 67(5):3009-3037, 2021. 1, 2, 8

William E. Leeb. Matrix denoising for weighted loss functions and heterogeneous signals. SIAM J. Math. Data Sci. , 3(3):987-1012, 2021. 2, 8

Jeffrey T. Leek. Asymptotic conditional singular value decomposition for high-dimensional genomic data. Biometrics , 67(2):344-352, 2011. 1

Thibault Lesieur, Florent Krzakala, and Lenka Zdeborová. Mmse of probabilistic low-rank matrix estimation: Universality with respect to the output channel. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 680-687, 2015. 2

Yue M. Lu. Householder dice: a matrix-free algorithm for simulating dynamics on Gaussian and random orthogonal ensembles. IEEE Trans. Inform. Theory , 67(12):8264-8272, 2021. 2

Clément Luneau, Jean Barbier, and Nicolas Macris. Mutual information for low-rank even-order symmetric tensor estimation. Inf. Inference , 10(4):1167-1207, 2021. 2

Pierre Mergny, Justin Ko, and Florent Krzakala. Spectral phase transition and optimal pca in block-structured spiked models. arXiv preprint arXiv:2403.03695 , 2024. 2

Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica , 70(2):583-601, 2002. 6, 30, 31

Léo Miolane. Fundamental limits of low-rank matrix estimation: the non-symmetric case. arXiv preprint arXiv:1702.00473 , 2017. 1, 2, 6, 17

Léo Miolane. Fundamental limits of inference: A statistical physics approach. Theses, Ecole normale supérieure - ENS PARIS ; Inria Paris, June 2019. 2, 17, 52

Marco Mondelli, Christos Thrampoulidis, and Ramji Venkataramanan. Optimal combination of linear and spectral estimators for generalized linear models. Foundations of Computational Mathematics , pages 1-54, 2021. 3

Marco Mondelli and Ramji Venkataramanan. Pca initialization for approximate message passing in rotationally invariant models. In Advances in Neural Information Processing Systems , volume 34, pages 29616-29629, 2021. 10

Andrea Montanari and Ramji Venkataramanan. Estimation of low-rank matrices via approximate message passing. Ann. Statist. , 49(1):321-345, 2021. 1, 3, 10

Andrea Montanari and Yuchen Wu. Fundamental limits of low-rank matrix estimation with diverging aspect ratios. arXiv preprint arXiv:2211.00488 , 2022. 1

Andrea Montanari and Yuchen Wu. Posterior sampling from the spiked models via diffusion processes. arXiv preprint arXiv:2304.11449 , 2023. 3

Raj Rao Nadakuditi. OptShrink: an algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage. IEEE Trans. Inform. Theory , 60(5):3002-3018, 2014. 2, 8

Boaz Nadler. Finite sample approximation results for principal component analysis: a matrix perturbation approach. Ann. Statist. , 36(6):2791-2817, 2008. 2

Aleksandr Pak, Justin Ko, and Florent Krzakala. Optimal algorithms for the inhomogeneous spiked wigner model. In Advances in Neural Information Processing Systems , volume 36, pages 76409-76424, 2023. 2

Dmitry Panchenko. The Sherrington-Kirkpatrick model . Springer Monographs in Mathematics. Springer, New York, 2013. 52

Henrik Pedersen, Sebastian Kozerke, Steffen Ringgaard, Kay Nehrke, and Won Yong Kim. k-t pca: Temporally constrained k-t blast reconstruction using principal component analysis. Magnetic Resonance in Medicine , 62(3):706-716, 2009. 1

S. Rangan. Generalized approximate message passing for estimation with random linear mixing. In IEEE International Symposium on Information Theory (ISIT) , 2011. 3

Sundeep Rangan, Philip Schniter, and Alyson K Fletcher. Vector approximate message passing. IEEE Transactions on Information Theory , 65(10):6664-6684, 2019. 3

Galen Reeves. Information-theoretic limits for the matrix tensor product. IEEE Journal on Selected Areas in Information Theory , 1(3):777-798, 2020. 2

R. Tyrrell Rockafellar. Convex analysis . Princeton Landmarks in Mathematics. Princeton University Press, Princeton, NJ, 1997. 30

Charles M. Stein. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics , 9(6):1135-1151, 1981. 52

Pei-Chun Su and Hau-Tieng Wu. Data-driven optimal shrinkage of singular values under high-dimensional noise with separable covariance structure. arXiv preprint arXiv:2207.03466 , 2022. 1, 2, 3

O. Tamuz, T. Mazeh, and S. Zucker. Correcting systematic effects in a large set of photometric light curves. Monthly Notices of the Royal Astronomical Society , 356(4):1466-1470, 02 2005. 1

Ramji Venkataramanan, Kevin Kögler, and Marco Mondelli. Estimation in rotationally invariant generalized linear models via approximate message passing. In International Conference on Machine Learning (ICML) , 2022. 3

Yihong Wu and Harrison H. Zhou. Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in O ( √ n ) iterations. Math. Stat. Learn. , 4(3-4):143220, 2021. 1

Kaylee Y Yang, Timothy LH Wee, and Zhou Fan. Asymptotic mutual information in quadratic estimation problems over compact groups. arXiv preprint arXiv:2404.10169 , 2024. 2

Xiaodong Yang, Buyu Lin, and Subhabrata Sen. Fundamental limits of community detection from multi-view data: multi-layer, dynamic and partially labeled block models. arXiv preprint arXiv:2401.08167 , 2024. 2

Ming Yuan. High dimensional inverse covariance matrix estimation via linear programming. J. Mach. Learn. Res. , 11:2261-2286, 2010. 10

Anru R. Zhang, T. Tony Cai, and Yihong Wu. Heteroskedastic PCA: algorithm, optimality, and applications. Ann. Statist. , 50(1):53-80, 2022. 2, 3, 8

Cun-Hui Zhang and Stephanie S. Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. J. R. Stat. Soc. Ser. B. Stat. Methodol. , 76(1):217-242, 2014. 8

Lixin Zhang. Spectral analysis of large dimentional random matrices . PhD thesis, National University of Singapore, 2007. 39

Yihan Zhang, Hong Chang Ji, Ramji Venkataramanan, and Marco Mondelli. Spectral estimators for structured generalized linear models via approximate message passing. arXiv preprint arXiv:2308.14507 , 2023. 3, 7, 39

Yihan Zhang, Marco Mondelli, and Ramji Venkataramanan. Precise asymptotics for spectral methods in mixed generalized linear models. arXiv preprint arXiv:2211.11368 , 2022. 3

Xinyi Zhong, Tianhao Wang, and Zhou Fan. Approximate message passing for orthogonally invariant ensembles: Multivariate non-linearities and spectral initialization. arXiv preprint arXiv:2110.02318 , 2021. 10

Yuchen Zhou and Yuxin Chen. Deflated heteropca: Overcoming the curse of ill-conditioning in heteroskedastic pca. arXiv preprint arXiv:2303.06198 , 2023. 3

Notation. All vectors are column vectors. The singular values of a matrix A ∈ R n × d (where n ≥ d without loss of generality) are denoted by σ 1 ( A ) ≥ · · · ≥ σ d ( A ) ≥ 0 and the corresponding left/right singular vectors are denoted by u 1 ( A ) , · · · , u d ( A ) ∈ S n -1 and v 1 ( A ) , · · · , v d ( A ) ∈ S d -1 . The (real) eigenvalues of a symmetric matrix B ∈ R d × d are denoted by λ 1 ( B ) ≥ · · · ≥ λ d ( B ) and the corresponding eigenvectors are denoted by v 1 ( B ) , · · · , v d ( B ) ∈ S d -1 (which will not be confused with the right singular vectors, whenever they are different, since we will never talk about both simultaneously for a square asymmetric matrix). We generally put overlines on capital letters to indicate a scalar random variable, e.g., X ∈ R , whose support is denoted by supp( X ) . The limit/liminf/limsup in probability are denoted by p-lim , p-liminf , p-limsup . The product distribution whose i -th ( i ∈ [ k ] ) marginal is given by P i is denoted by P 1 ⊗···⊗ P k , with the shorthand P ⊗ k when all P i 's are equal to P . The gradient of f : R n → R , or with abuse of notation, the Jacobian matrix of F : R n → R d are denoted by ∇ f ∈ R n , ∇ F ∈ R d × n . The partial derivative of f ( x 1 , · · · , x n ) with respect to x i is denoted by either ∂ ∂x i f ( x 1 , · · · , x n ) or ∂ i f ( x 1 , · · · , x n ) . All log and exp are to the base e . We use the standard notation of sup( S ) , inf( S ) for a subset S ⊂ R . We generally use C > 0 to denote a sufficiently large constant independent of n, d . Its dependence on other parameters will be specified, though its value may change across passages. We use the standard big O notation.

A Proof of Proposition 4.1

We eliminate q u and write a fixed point equation only involving q v :

Denote the RHS by f ( q v ) . Recall that we are only interested in non-negative solutions ( q u , q v ) . So let us restrict attention on f to the domain R ≥ 0 . We have f (0) = 0 and

It then becomes evident that a non-trivial fixed point q v > 0 exists if and only if f ′ (0) > 1 and in this case, the non-trivial fixed point is unique.

Finally, by the first equation in (4.5), there is a non-trivial fixed point q u if and only if there is a non-trivial fixed point q v , which completes the proof.

B Auxiliary Gaussian channel

We formally introduce here the auxiliary model mentioned in Section 4. Consider a Gaussian channel with blocklength n , input x ∗ , output Y , anisotropic Gaussian noise Σ 1 / 2 Z and SNR γ :

where

By similar derivations as in Section 4, the posterior distribution of x ∗ given Y can be written as

where the Hamiltonian and the partition function are

With P = N (0 , 1) , Z n ( γ ) becomes a Gaussian integral that can be computed as below using Proposition G.1:

Therefore, by Proposition G.2,

The above functional is nothing but ψ Σ ( γ ) introduced in (4.15) which will play an important role in characterizing the free energy of the original model (4.1).

C Proof of Theorem 4.4

Before diving into the proof, we make further notation adjustments for the ease of applying the interpolation argument. Specifically, we will henceforth assume γ = 1 by incorporating the actual value of γ into the prior distributions P, Q ,

This is obviously equivalent to the previous setting. So we can drop the dependence on γ and write MMSE n , Z n , F n for MMSE n ( γ ) , Z n ( γ ) , F n ( γ ) defined in (4.2), (4.13) and (4.14).

We will also assume that Ξ , Σ are diagonal. This is without loss of generality since the Gaussianity of P, Q, ˜ W ensures that both the prior distributions and the noise matrix are rotationally invariant. Furthermore, we truncate P, Q so that they are supported on [ -K,K ] for a constant K > 0 . The approximation error in the free energy due to truncation can be made arbitrarily small if K is sufficiently large, since the free energy is pseudo-Lipschitz in the prior distribution with respect to the Wasserstein2 metric.

The proof follows an interpolation argument [10, 52, 53] with suitable modifications to take care of the noise heteroscedasticity featured by the covariances Ξ , Σ . To start with, define the interpolating models:

Define the free energy as

where q 1 ( t ) , q 2 ( t ) ≥ 0 are to be determined and

By definition, Y 0 = Y is the model that we would like to understand, and Y u t , Y v t are instances of Gaussian channels in (B.1) whose free energy we have already understood (see (B.2)). The idea is that Y t serves as a path parametrized by t ∈ [0 , 1] from the original model Y to the target models ( Y u 1 , Y v 1 ) . The crux of the interpolation argument lies in showing that Y t and ( Y u t , Y v t ) are equivalent (at the level of free energy) along the path.

To study the interpolating models ( Y t , Y u t , Y v t ) , define the Hamiltonian

Then the posterior distribution of ( u ∗ , v ∗ ) given ( Y t , Y u t , Y v t ) is

Let the partition function be

Define the free energy as where we recall the notation

The Gibbs bracket ⟨·⟩ n,t denotes the expectation with respect to the posterior distribution in (C.3):

for any g : R n × R d → R such that the expectation exists. That is,

We will also use the notation ⟨·⟩ n for the Gibbs bracket with respect to the original posterior d P ( u, v | Y ) in (4.11).

Lemma C.1. Consider f n ( t ) defined in (C.5) with t ∈ { 0 , 1 } . Assume that q 1 (0) , q 2 (0) satisfy

Then we have

where lim K →∞ o K = 0 .

Proof. To show the first statement (C.8), let us control f n (0) -F n . Denoting

and recalling the Gibbs bracket notation ⟨·⟩ n , we have

where the outer expectation is over all randomness in u ∗ , v ∗ , Z u , Z v . The second equality above follows since

We will derive double-sided bounds on f n (0) -F n .

To upper bound it, use Jensen's inequality E [log( · )] ≤ log E [ · ] on the partial expectation over Z u , Z v in (C.10):

where the equality is legit since ⟨·⟩ n does not depend on Z u , Z v . By the Gaussian integral formula (Proposition G.1), the inner expectation equals

Replacing the Gibbs bracket with max , we obtain an upper bound:

where the last inequality holds for all sufficiently large n by (3.3) and n ≍ d .

To lower bound f n -F n , we use Jensen's inequality again but this time on the Gibbs bracket in (C.10):

Since ( Z u , Z v ) ∼ N (0 n , I n ) ⊗N (0 d , I d ) , the inner expectation equals

So

Using this, we obtain a lower bound on f n -F n by replacing the Gibbs bracket on the RHS of (C.12) with -max |·| :

where the last inequality holds for all sufficiently large n by (3.3) and n ≍ d . Combining (C.11) and (C.13) gives the first result (C.8).

We then prove the second statement (C.9). Since f n is pseudo-Lipschitz as a function of the priors, up to a term o K that vanishes as K →∞ uniformly over n , it suffices to ignore the truncation at K and assume P = N (0 , γ ) , Q = N (0 , 1) . From the definition (C.4) of Z n,t , we have

where in the second equality, we use the Gaussian integral formula (Proposition G.1). Therefore

verifying the identity (C.9). In the second equality, we have used Proposition G.2.

Lemma C.2 (Free energy variation) . For all t ∈ (0 , 1) ,

Proof. From the definitions (C.4) and (C.5), we compute

By the definition (C.2), the time derivative of the Hamiltonian is

The expectation of ⟨ ∂ t H n,t ⟩ n,t can be computed using the Stein's lemma (Proposition G.7). Indeed, let us consider the term

where the last step is by the Nishimori identity (Proposition G.4). So the first line of (C.15) upon taken the Gibbs bracket and the expectation becomes

Similar cancellations happen for the second and third lines of (C.15). Putting them together, we obtain

which is the same as (C.14) with the parentheses opened up.

In what follows, our strategy is:

Show that 〈 ˜ u ⊤ ˜ u ∗ 〉 n,t is concentrated around its mean E [ 〈 ˜ u ⊤ ˜ u ∗ 〉 n,t ] ;

Choose q 2 ( t ) to be the solution to

Once Items 1 and 2 are done, we then have

where the first line above uses (C.8) in Lemma C.1; the second line uses (C.9) in Lemma C.1 and Lemma C.2; the third line uses Items 1 and 2. This will almost lead to the desired characterization of the free energy F n in Theorem 4.4:

Note that ϕ t ( q 1 , q 2 ) also depends on n, u ∗ , v ∗ , Z, Z u , Z v , and E [ ϕ t ( q 1 , q 2 )] = f n ( t ) where the expectation is over ( u ∗ , v ∗ , Z, Z u , Z v ) distributed according to (C.1).

Consider the function

Lemma C.3 (Free energy concentration) . Fix a constant M > 0 . There exists a constant C > 0 depending only on K,M,α, Ξ , Σ such that for any t ∈ [0 , 1] , 0 ≤ q 1 ( t ) , q 2 ( t ) ≤ M and sufficiently large n ,

Proof. Fix u ∗ , v ∗ . Consider ϕ t ( q 1 , q 2 ) as a function of ( Z, Z u , Z v ) . Then

where C > 0 is a constant depending only on α, M, Ξ , Σ , K . The penultimate line is by CauchySchwarz and the last line holds for all sufficiently large n by (3.3) and n ≍ d . Then by the Gaussian Poincaré inequality (Proposition G.8),

The above result holds for any fixed u ∗ , v ∗ . We then verify that E Z,Z u ,Z v [ ϕ t ( q 1 , q 2 )] has bounded difference as a function of u ∗ , v ∗ . We do so by bounding the derivatives of E Z,Z u ,Z v [ ϕ t ( q 1 , q 2 )] with respect to u ∗ i , v ∗ j for any i ∈ [ n ] , j ∈ [ d ] . We have

and a similar expression holds for the derivative with respect to v ∗ j . Recall the definition of H n,t from (C.2). We have

where we have used the fact that Ξ is diagonal. Therefore,

where C > 0 is a constant depending only on α, M, K, Ξ , Σ . The last inequality holds for all sufficiently large n . A similar bound holds for ∂ ∂v ∗ j H n,t (Ξ -1 / 2 u, Σ -1 / 2 v ; q 1 , q 2 ) . This, by (C.18), implies that E Z,Z u ,Z v [ ϕ t ( q 1 , q 2 )] as a function of ( u ∗ , v ∗ ) satisfies the bounded difference property with c i = C/n (see Proposition G.9). So by Proposition G.9,

Finally using (C.17) and (C.19) and the triangle inequality,

E [ | ϕ t ( q 1 , q 2 ) -E [ ϕ t ( q 1 , q 2 )] | ]

concluding the proof.

Suppose a, b ≥ 0 are constants. With q 1 = s n a 2 , q 2 = s n b 2 , we can write H n,t in (C.2) as

where

Fix A ≥ 2 . Define

Denote by ⟨ g ( u ∗ , v ∗ ) ⟩ n,a,b the conditional expectation of g ( u ∗ , v ∗ ) given

where the expectation is with respect to the distribution in (C.1).

Corollary C.4. Let s n = n -1 / 32 and A ≤ √ M/s n -1 / 2 for a constant M > 0 independent of n . Then there exists a constant C > 0 depending only on K,α, Ξ such that for all sufficiently large n ,

Proof. Note that if b = 0 , t = 1 , it holds that H n, 1 ( ˜ u, ˜ v ; s n a 2 , 0) = H u n,a ( ˜ u ) and ϕ 1 ( s n a 2 , 0) = s n ϕ u ( a ) . The conclusion then follows immediately from Lemma C.3 since by the assumption on A , q 1 = s n a 2 ∈ [0 , M ] for any 1 / 2 ≤ a ≤ A +1 / 2 .

Lemma C.5. Let s n = n -1 / 32 . For all A ≥ 2 ,

where C > 0 only depends on α, Ξ , K .

Proof. By the triangle inequality,

We will bound the two terms on the RHS separately. We first bound

The first two derivatives of ϕ u are

Since there exists C > 0 depending only on α, Ξ , K such that for all sufficiently large n ,

the second result (C.24b) above implies that for all sufficiently large n ,

Consequently,

To proceed, we compute for any a ,

which is by (C.24a). By definition,

whose expectation is therefore given by

Using Stein's lemma (Proposition G.7), the middle term is equal to

Therefore, and

where C depends only on α, Ξ , K .

Using the last inequality, we can further upper bound the RHS of (C.26) by CA ns n for some C depending only on α, Ξ , K . Putting this back to (C.23), we get

since A ≥ 2 .

Now it remains to bound the second term on the RHS of (C.22) which can be written as

using (C.24a). To further bound the RHS, consider the following two functions

They are both differentiable and convex for all sufficiently large n since their second derivatives are non-negative by (C.25) and (C.24b). Applying Proposition G.10 with the above two functions, taking the expectation and using the triangle inequality, we have that for any 1 ≤ a ≤ A, 0 < a ′ ≤ 1 / 2 ,

Then

where the last step is by (C.27) and (C.30), and C depends only on α, Ξ , K . Using this in (C.33), we obtain

and the RHS is minimized by a ′ = √ ξ u n ( s n ) which lies in the interval (0 , 1 / 2] for all sufficiently large n due to Corollary C.4. Using this result in (C.32) and integrating over a , we have

since A ≥ 2 .

Finally, combining (C.22), (C.31) and (C.34) proves the lemma.

Lemma C.6. There exists C > 0 depending only on α, K, Ξ such that for any A ≥ 2 ,

where ˜ u (1) = Ξ -1 / 2 u (1) , ˜ u (2) = Ξ -1 / 2 u (2) with u (1) , u (2) being two i.i.d. copies from the conditional law of u ∗ , v ∗ given (C.21) .

Proof. Since supp( P ) , supp( Q ) , ∥ Ξ ∥ -1 2 are all bounded, by the triangle inequality,

On the other hand, let us compute (C.35) on the LHS of the above inequality. Recall from (C.29) in the proof of Lemma C.5 that

Similar to (C.28), we have

Using Stein's lemma and following similar derivations leading to (C.29), the second term on the RHS equals

Therefore,

Putting (C.37) and (C.38) together and using Nishimori identity (Proposition G.4), we have that (C.35) equals

So by the inequality (C.36), for any a ≥ 1 ,

Integrating over a ∈ [1 , A ] and invoking Lemma C.5 concludes the proof.

Lemma C.7 (Overlap concentration) . Let R 1 , R 2 : [0 , 1] × R 2 > 0 → R ≥ 0 be two continuous bounded functions such that their partial derivatives with respect to the second and third arguments are continuous and non-negative. Let s n = n -1 / 32 . For ( ε 1 , ε 2 ) ∈ [1 , 2] 2 , slightly abusing notation, let q 1 ( · , ε 1 , ε 2 ) , q 2 ( · , ε 1 , ε 2 ) be the unique solution to

Then there exists a constant C > 0 depending only on K , α , ∥ R 1 ∥ ∞ , ∥ R 2 ∥ ∞ , Ξ such that for every t ∈ [0 , 1] ,

Proof. The existence and uniqueness of the solution to the Cauchy problem (C.39) is a direct consequence of the Cauchy-Lipschitz theorem [37, Theorem 3.1, Chapter V]. For any t ∈ [0 , 1] , the function Q t ( ε 1 , ε 2 ) = ( q 1 ( t, ε 1 , ε 2 ) , q 2 ( t, ε 1 , ε 2 )) is a C 1 -diffeomorphism. Its Jacobian determinant is given by the Liouville formula [37, Corollary 3.1, Chapter V] and can be lower bounded as

since the partial derivatives are non-negative by assumptions.

We then view the RHS below as a function of q 1 , q 2 and denote it by

Denote Ω t = Q t ([1 , 2] 2 ) /s n and M := max {∥ R 1 ∥ ∞ , ∥ R 2 ∥ ∞ } + 2 . Since R 1 , R 2 ≥ 0 by assumptions, q 1 , q 2 are non-decreasing in t by (C.39). So for i ∈ { 1 , 2 } , and any t ∈ [0 , 1] and ( ε 1 , ε 2 ) ∈ [1 , 2] 2 ,

We obtain the relation Ω t ⊂ [1 , M/s n ] 2 for any t ∈ [0 , 1] .

Next, using the change of variable ( r 1 , r 2 ) = Q t ( ε 1 , ε 2 ) /s n , we have

where the last step is by (C.40). Further applying the change of variable r 1 = a 2 , we have that for all r 2 ≥ 1 ,

Recalling V from (C.41), we recognize that

where ⟨·⟩ n,a,b is defined in (C.21) and the second equality is by Nishimori identity (Proposition G.4). Since √ M/s n ≥ 2 for all sufficiently large n , applying Lemma C.6, we get

where C > 0 depends only on α, K, Ξ and ξ u n ( s n ) is given in (C.20) with A = √ M/s n . Using this back in (C.43) and then in (C.42), we obtain

Corollary C.4 guarantees that ξ u n ( s n ) ≤ C s n √ n for some C > 0 depending only on α, K, Ξ . By the choice of s n , we can finally upper bound (up to a positive constant depending only on α, K, Ξ , M ) the RHS above by

1

s

2

n

1

ns

n

√

s

1

n

n

s

1

2

.

5

n

n

2

1

/

for all sufficiently large n , which completes the proof.

Recalling ⟨·⟩ n,t defined in (C.6), let us identify E [ 〈 ˜ u ⊤ ˜ u ∗ 〉 n,t ] as a function of ( t, q 1 , q 2 ) :

Note that ∆ is continuous, non-negative (by Nishimori identity) on [0 , 1] × R 2 ≥ 0 and bounded by K 2 . Its partial derivatives with respect to the second and third arguments are continuous and non-negative, since the correlation between ˜ u ∗ and ⟨ ˜ u ⟩ n,t is a non-decreasing function of the SNRs q 1 , q 2 .

Lemma C.8 (Fundamental sum rule) . In the setting of Lemma C.7, for ( ε 1 , ε 2 ) ∈ [1 , 2] 2 , let q 1 ( t, ε 1 , ε 2 ) , q 2 ( t, ε 1 , ε 2 ) be the solution to (C.39) with R 2 = ∆ defined in (C.44) . Then we have

Proof. Fix ( ε 1 , ε 2 ) ∈ [1 , 2] 2 . By the choice R 2 = ∆ ,

Plugging this into (C.14), integrating over ( ε 1 , ε 2 ) ∈ [1 , 2] 2 and applying Lemma C.7, we have

where o (1) → 0 as n →∞ , uniformly over t . By Lemma C.1, we conclude

as desired. Here the first and last equalities are by (C.8) and (C.9), respectively.

Finally, we prove a pair of matching upper and lower bounds, completing the proof of Theorem 4.4.

Lemma C.9 (Lower bound) . Let s n = n -1 / 32 and R 2 = ∆ . Then

Proof. Fix an arbitrary q v ≥ 0 . Let R 1 = q v . Then q 1 ( t, ε 1 , ε 2 ) = s n ε 1 + tq v and Lemma C.8 gives

where the second line holds since ψ Ξ is Lipschitz and q 2 (0 , ε 1 , ε 2 ) = s n ε 2 = o (1) . This completes the proof since the above lower bound holds for all q v ≥ 0 .

(

)

√

+

√

·

4

= 2

·

n

-

11

/

64

2

·

n

-

1

/

8

,

Lemma C.10 (Upper bound) . Let s n = n -1 / 32 and R 2 = ∆ . Then

Proof. We apply Lemma C.8 with

Since ψ Ξ is Lipschitz and convex,

and similarly

where

With the choice of R 1 in (C.45) and R 2 = ∆ , the ODE in Lemma C.7 gives

which corresponds to the criticality condition of G with respect to q u :

Since ψ Σ is convex and -α 2 q u q v is linear in q 2 , we have that G is convex in q u . Therefore

which completes the proof.

D Proofs of Theorem 4.2 and Corollary 4.3

D.1 Proof of (4.7)

We compute the derivative of F n ( γ ) :

where the last step follows similar calculations in (C.16). Since F n ( γ ) → sup q v inf q u F ( q u , q v ) as n →∞ , we have F ′ n ( γ ) → ∂ ∂γ sup q v inf q u F ( q u , q v ) . To compute the RHS, note that

and the value of the infimum does not depend on γ . Therefore, we have

where first equality is by the envelope theorem from [51, Corollary 4] and the last equality follows since the extremizers q ∗ u , q ∗ v solve a pair of equations in (4.16).

On the other hand, we relate F ′ n ( γ ) to the MMSE (4.2) as follows:

where the last step is by Nishimori identity (Proposition G.4). Combining the above with (D.1) and (D.2), we conclude

as claimed.

D.2 Proof of (4.8)

Recall Y from (4.1) and define for some γ ′ ≥ 0 ,

where Z ′ ∈ R n × n is a symmetric random matrix independent of u ∗ , v ∗ with Z ′ i,i i . i . d . ∼ N (0 , 2) and Z ′ i,j i . i . d . ∼ N (0 , 1) for all 1 ≤ i < j ≤ n . By similar derivations as before, the free energy associated with ( Y, Y ′ ) is given by

where H n is given in (4.12). Denote by ⟪ · ⟫ n the Gibbs bracket with respect to the corresponding Hamiltonian. Let

where f ∗ denotes the monotone conjugate of a convex non-decreasing function f : R ≥ 0 → R ; see Definition G.1. Basic properties of monotone conjugate can be found in [67, §12].

The following lemma, proved in Appendix D.3, characterizes the high-dimensional limit of F n ( γ, γ ′ ) . Lemma D.1. For all γ, γ ′ ≥ 0 ,

Let us show how (4.8) can be derived from Lemma D.1.

Proof of (4.8) . Let

Following similar derivations as in Appendix D.1, one can verify the following two identities:

Therefore, the MMSE in (4.3) can be written as

By Lemma D.1 and Proposition G.11,

The envelope theorem from [51, Corollary 4] allows us to compute the RHS:

where ( q ∗ u , q ∗ v ) are the maximizer of

where the equality is by Proposition G.12. Putting the above together, we have

By a symmetric argument, we also have

To find an upper bound on MMSE u n ( γ ) , note that by (4.7) and (D.3):

This combined with (D.7) and (D.8) implies

which concludes the proof in view of the relation

D.3 Proof of Lemma D.1

We assume γ = γ ′ = 1 by formally absorbing them into P, Q :

so that we can drop the dependence on γ, γ ′ in notation such as F n , F . We then truncate P, Q at a sufficiently large constant K > 0 so that they have bounded supports.

Recall Y from (4.1) and define for r ≥ 0 , Y ′′ := √ ru ∗ + Ξ 1 / 2 Z ′′ where Z ′′ ∼ N (0 n , I n ) is independent of everything else. The free energy ˜ F n associated with ( Y, Y ′′ ) is

where H n is given in (4.12). A straightforward adaptation of the proof of Theorem 4.4 yields a characterization of the limit of ˜ F n . Let

Proof. To obtain the result, we execute the interpolation argument as in the proof of Theorem 4.4 on the Hamiltonian of the following interpolating models:

All steps in the proof of Theorem 4.4 carry over.

The above lemma allows us to derive the following auxiliary characterization of F n .

Lemma D.3.

Proof. Let r : [0 , 1] → R ≥ 0 be a differentiable function. For t ∈ [0 , 1] , consider ( Y, Y ′ t , Y ′′ t ) where Y is given in (4.1) and Y ′ t , Y ′′ t are defined as

with Z ′′ ∼ N (0 n , I n ) independent of everything else.

Similar to (C.5) and (C.6), denote by

the free energy associated with ( Y, Y ′ t , Y ′′ t ) and by ⟪ · ⟫ n,t the conditional expectation with respect to the corresponding Gibbs measure. The rest of the proof follows the skeleton of Theorem 4.4 and we only highlight the differences.

Parallel to Lemma C.1, we have that if r (0) ≥ 0 and lim n →∞ r (0) = 0 , then

The analogue of Lemma C.2 gives

We now show a pair of matching lower and upper bounds on F n . First comes the lower bound. Using (D.11) with r ( t ) = rt for a constant r ≥ 0 , we have

By Lemma D.2, this implies the lower bound

Next we show a matching upper bound. Let ε ∈ [1 , 2] and slightly abusing notation, denote by r ( t ; ε ) the solution to

where s n = n -1 / 32 . The analogue of Lemma C.7 gives

for a constant C > 0 independent of n . Using (D.11) with r ( t ; ε ) , we have

where the second equality is by (D.12) and the penultimate inequality holds since ˜ F n ( · ) is convex and non-decreasing. Passing to the limit, we obtain the upper bound

as desired.

To establish Lemma D.1, it remains to verify that the RHSs of (D.5) and (D.10) are equal. We need the following lemma.

Lemma D.4. Let f, g : R ≥ 0 → R be non-decreasing, lower semi-continuous, convex functions with finite f (0) , g (0) and monotone conjugates f ∗ , g ∗ (see Definition G.1). Then

Proof. Writing f r ( x ) = f ( x + r ) and using Proposition G.12, we have

where we have used the fact that f ∗∗ = f . Therefore,

as claimed.

Applying Lemma D.4 immediately finishes the proof of Lemma D.1.

Proof of Lemma D.1. By Lemma D.4 and the definition (D.9),

where we have used the fact that for a, b ≥ 0 , the monotone conjugate of g ( x ) := bf ( ax ) is g ∗ ( x ) = bf ∗ ( x/ ( ab )) .

D.4 Proof of Corollary 4.3

Denote M := E [ u ∗ v ∗⊤ ∣ ∣ ∣ Y ] . By the Nishimori identity (Proposition G.4),

Theorem 4.2 and (D.13) imply that

which, by Proposition 4.1, is positive if and only if (4.6) holds.

We first show (4.10) assuming (4.6). Using assumption (3.3), super-multiplicativity of σ min ( · ) , and the fact that ∥ BC ∥ F ≥ ∥ B ∥ F σ min ( C ) , we have

which is positive by (D.15). This combined with (D.14) implies (4.10).

We then show (4.10) with < replaced with = , assuming that (4.6) is reversed. Using assumption (3.3), sub-multiplicativity of spectral norm, and the fact that ∥ BC ∥ F ≤ ∥ B ∥ F ∥ C ∥ 2 , we have

which is 0 by (D.15). This combined with (D.14) finishes the proof of the corollary for estimating u ∗ v ∗⊤ . The proofs for estimating u ∗ u ∗⊤ , v ∗ v ∗⊤ are similar and omitted.

E Analysis of the spectral estimator

E.1 Bayes-AMP

We propose an AMP algorithm that operates on Ξ -1 A Σ -1 and maintains a pair of iterates u t ∈ R n , v t +1 ∈ R d for every t ≥ 0 . Specifically, given for every t ≥ 0 a pair of denoising functions 2 g t : R n → R n , f t +1 : R d → R d , the iterates are initialized at ˜ u -1 = 0 n and some ˜ v 0 ∈ R d of user's choice, and are updated for every t ≥ 0 according to the following rules:

where ∇ g t ( u t ) ∈ R n × n , ∇ f t +1 ( v t +1 ) ∈ R d × d denote the Jacobians of g t , f t +1 at u t , v t +1 , respectively. For any fixed t ≥ 0 , the n, d → ∞ limit of the iterates u t , v t +1 can be described by a deterministic recursion known as the state evolution. To define the latter, we need a sequence of preliminary definitions.

First, define random vectors ( U ∗ , W U, 0 , W U, 1 , · · · , W U,t ) ∈ ( R n ) t +2 and ( V ∗ , W V, 1 , W V, 2 , · · · , W V,t +1 ) ∈ ( R d ) t +2 with joint distributions specified below:

where we recall that for A ∈ R m × n , B ∈ R p × q , their Kronecker product is

The covariance matrices Φ t = (Φ r,s ) 0 ≤ r,s ≤ t , Ψ t +1 = (Ψ r +1 ,s +1 ) 0 ≤ r,s ≤ t ∈ R ( t +1) × ( t +1) are given by the ( t + 1) × ( t + 1) principal minors of two infinite-dimensional symmetric matrices Φ := (Φ r,s ) r,s ≥ 0 , Ψ := (Ψ r +1 ,s +1 ) r,s ≥ 0 whose elements are in turn given recursively below:

Furthermore, for t ≥ 0 , σ t , τ t +1 > 0 are defined as

With the above definitions, note that each W U,t , W V,t +1 is marginally distributed as N (0 n , I n ) , N (0 d , I d ) , respectively.

Next, define two sequences of deterministic scalars ( µ t , ν t +1 ) t ≥ 0 :

where f 0 is determined by the initializer ˜ v 0 ; see Assumption (A1) below.

With these, for t ≥ 0 , define random vectors

Finally, we need the notion of (uniformly) pseudo-Lipschitz functions.

Definition E.1 (Pseudo-Lipschitz functions) . A function ϕ : R k × m → R ℓ × m is called pseudoLipschitz of order j ≥ 1 if there exists L > 0 such that

for every x, y ∈ R k × m .

We will consider sequences of functions ϕ i : R k i × m → R ℓ i × m indexed by i →∞ though the index i is often suppressed. A sequence of functions ( ϕ i : R k i × m → R ℓ i × m ) i ≥ 1 (with increasing dimensions ( k i ) i ≥ 1 , ( ℓ i ) i ≥ 1 ) is called uniformly pseudo-Lipschitz of order j if there exists a constant L such that for every i ≥ 1 , (E.7) holds.

The assumptions below are imposed on the initializer ˜ v 0 and denoising function ( g t , f t +1 ) t ≥ 0 .

(A1) ˜ v 0 is independent of ˜ W but may depend on v ∗ . 3 Assume that

exist and are finite. There exists a uniformly pseudo-Lipschitz function f 0 : R d → R d of order 1 such that

and for every uniformly pseudo-Lipschitz function ϕ : R d → R d of finite order, the following two limits exist, are finite and equal:

Let ˜ ν ∈ R , ˜ τ ∈ R ≥ 0 . For any s ≥ 1 ,

exists and is finite, where W V ∼ N (0 d , I d ) is independent of V ∗ . (A2) Let ˜ ν ∈ R , and T ∈ R 2 × 2 be positive definite. For any r, s ≥ 1 ,

exist and are finite, where ( N,N ′ ) ∼ N (0 2 d , T ⊗ I d ) is independent of V ∗ . Let µ ∈ R , and S ∈ R 2 × 2 be positive definite. For any r, s ≥ 0 ,

˜

exist and are finite, where ( M,M ′ ) ∼ N (0 2 n , S ⊗ I n ) is independent of U ∗ .

We now give the state evolution result for the AMP in (E.1), which is proved in Appendix F.

Proposition E.1 (State evolution for AMP in (E.1)) . For every t ≥ 0 , let ( g t : R n → R n ) n ≥ 1 and ( f t +1 : R d → R d ) d ≥ 1 be uniformly pseudo-Lipschitz of finite order subject to Assumption (A2) . Consider the AMP iteration in (E.1) defined by ( g t , f t +1 ) t ≥ 0 and initialized at ˜ u -1 = 0 n and some ˜ v 0 ∈ R d subject to Assumption (A1) . For any fixed t ≥ 0 , let ( ϕ : R ( t +2) n → R ) n ≥ 1 and ( ψ : R ( t +2) d → R ) d ≥ 1 be uniformly pseudo-Lipschitz of finite order. Then,

where ( U s , V s +1 ) 0 ≤ s ≤ t are defined in (E.6) .

Given U t , V t +1 , the Bayes-optimal (in terms of mean square error) choice of ( g t , f t +1 ) t ≥ 0 is given by the conditional expectations. Specifically, for any t ≥ 0 and u ∈ R n , v ∈ R d ,

We call (E.1) with g t = g ∗ t , f t +1 = f ∗ t +1 the Bayes-AMP.

If P = Q = N (0 , 1) , by (E.2) and (E.6), ( U ∗ , U t ) and ( V ∗ , V t +1 ) are jointly Gaussian with mean zero and covariance:

respectively. Therefore using Proposition G.5, g ∗ t , f ∗ t +1 admit the following explicit formulas:

Under the above choice, the state evolution recursion for µ t , σ t , ν t +1 , τ t +1 in (E.4) and (E.5) becomes: for all t ≥ 1 ,

where we have used the definitions (E.6) of U t , V t +1 , the joint distribution (E.2) of ( U ∗ , W U,t ) , ( V ∗ , V t +1 ) , the convergence of the empirical spectral distributions of Σ , Ξ , and Proposition G.2.

Inspecting the expressions, we realize that

This allows us to only track the recursion of µ t , ν t +1 :

Thus, the fixed point ( µ ∗ , ν ∗ ) of the above recursion must satisfy:

Note that upon a change of variable

the fixed point equation (E.13) coincides with that in the characterization of the free energy; see (4.5).

E.2 Spectral estimator from Bayes-AMP

Under Gaussian priors, the Bayes-AMP algorithm specified by (E.1) and (E.10) naturally suggests a spectral estimator with respect to a matrix which is a non-trivial transformation of A . In what follows, we provide a heuristic derivation of this spectral estimator. Its performance guarantee (Theorem 5.1) will be proved in Appendix E.4.

Suppose, informally, that µ t , σ t , ν t +1 , τ t +1 , u t , v t +1 , c t , b t +1 converge (under the sequential limits n →∞ , t →∞ ) to µ ∗ , σ ∗ , ν ∗ , τ ∗ , u, v, c ∗ , b ∗ , respectively, in the sense that, e.g.,

Recall that ( µ ∗ , ν ∗ ) solves the fixed point equation (E.13), and from (E.12), the following relation holds:

So denoting

by the design of g ∗ t , f ∗ t +1 in (E.10), we have that ˜ u t , ˜ v t +1 converge to

respectively, and the limiting Onsager coefficients b ∗ , c ∗ are given by:

At the fixed point of (E.1), we have

Upon rearrangement, (E.18) is equivalent to

where

We further introduce the notation:

so that (E.19) can be rewritten as

or

The key observation is that this is a pair of singular vector equations for the matrix

with respect to left/right singular vectors (up to rescaling)

and singular value 1 . Using the definitions (E.16), (E.20) and (E.21), we verify that the two expressions of A ∗ in (5.3) and (E.22) are equal.

By the state evolution result (Proposition E.1), u, v behave (in the sense of (E.8)) as

are effective estimates of u ∗ , v ∗ . Simple algebra reveals that the above vectors, upon suitable rescaling, are precisely ̂ u, ̂ v in (5.4).

E.3 Right edge of the bulk

Before proceeding with the proof of Theorem 5.1, we provide a characterization of σ 2 ( A ∗ ) , i.e., the right edge of the bulk of the spectrum of A ∗ . This bulk is related to the spectrum of the non-spiked random matrix

We first present a characterization of σ 1 ( W ∗ ) and then relate it to σ 2 ( A ∗ ) . Define random variables:

Define functions c, s : (sup supp(Ξ ∗ ) , ∞ ) → (0 , ∞ ) as

Define the implicit function β : (sup supp(Ξ ∗ ) , ∞ ) → (0 , ∞ ) as, for any α ∈ (sup supp(Ξ ∗ ) , ∞ ) , the unique solution in ( s ( α ) , ∞ ) to

(The existence and uniqueness of the solution is easy to see.) Next, define ψ : (sup supp(Ξ ∗ ) , ∞ ) → (0 , ∞ ) as ψ ( α ) = αβ ( α ) . It is known that ψ is differentiable and the set of its critical points is a nonempty finite set [79]. Let α ◦ ∈ (sup supp(Ξ ∗ ) , ∞ ) be the largest critical point, i.e.,

Finally, denote

The characterization of σ 1 ( W ∗ ) requires an extra technical assumption on the random variables Ξ , Σ , which is the same as in [79].

(A3) For any c > 0 ,

where s := c · sup supp(Σ ∗ ) . Furthermore,

Lemma E.2. Let Assumption (A3) hold. Consider W ∗ defined in (E.24) . Then, we have

Proof. Note that W ∗ W ∗⊤ is a separable covariance matrix. Its largest eigenvalue is characterized in [25]. The explicit formulas we need are due to [79]. To apply their results, one simply observes that the covariances (as in the context of separable covariance matrices) of W ∗ are

whose limiting spectral distributions are given by the distributions of Ξ ∗ , Σ ∗ in (E.25).

Lemma E.3. Consider A ∗ defined in (5.3) . Then

Proof. By Weyl's inequality, σ 3 ( W ∗ ) ≤ σ 1 ( A ∗ ) ≤ σ 1 ( W ∗ ) . We have already shown in Lemma E.2 that σ 1 ( W ∗ ) converges to σ ∗ 2 . The almost sure weak convergence of the empirical spectral distribution of W ∗ [78, Theorem 1.2.1] implies that σ 3 ( W ∗ ) (and indeed σ k ( W ∗ ) for any constant k relative to n, d ) must also converge to the same limit σ ∗ 2 .

E.4 Proof of Theorem 5.1

We suppose throughout the proof that the condition (5.1) holds. Then, by Proposition 4.1 and the change of variable (E.14), the fixed point equation (E.13) admits a unique non-trivial solution ( µ ∗ , ν ∗ ) ∈ R 2 > 0 . Construct matrices F, G as in (E.16) using such µ ∗ , ν ∗ . Define also the random variables

whose distributions are the limiting spectral distributions of G,F , respectively.

Now consider the denoising functions:

With this choice, the AMP iteration (E.1) becomes

where

Note that as n →∞ , b, c converge to b ∗ , c ∗ in (E.17).

Recall from (E.15) the definition of σ ∗ , τ ∗ . We initialize (E.27) with

where w ∼ N (0 d , I d ) is independent of everything else. Accordingly, one can take f 0 in Assumption (A1) to be f 0 ( v ) = ν ∗ F Σ -1 v . Under the above AMP initializer, the state evolution initializers in (E.3) and (E.5) specialize to

where the last equalities for both chains of computation are by (E.11). Since the parameters µ t , σ t are initialized at the non-trivial fixed point ( µ 0 , σ 0 ) = ( µ ∗ , σ ∗ ) , the state evolution recursion (E.11) will stay at the fixed point ( µ t , σ t , ν t +1 , τ t +1 ) = ( µ ∗ , σ ∗ , ν ∗ , τ ∗ ) across all t ≥ 0 .

Lemma E.4. Let

where ˜ F, ˜ G are defined in (E.21) . Suppose the condition (5.1) holds. Then

and

where A ∗ is defined in (E.22) .

Proof. Denoting

for any t ≥ 1 and using the notation ̂ F, ̂ G in (E.20), we have from (E.27) that

where

Let us focus on ̂ u t and only prove the first equality in (E.29). The proof of the second one is similar and will be omitted. Eliminating ̂ v t from (E.32) gives

Unrolling this recursion for s steps, we get:

Taking 1 n ∥·∥ 2 2 on both sides of (E.34) and take the sequential limits of n →∞ , t →∞ , s →∞ , we have the left hand side:

Next, we have that where

Recalling the notation ˜ F, ˜ G from (E.21) and multiplying ˜ G -1 / 2 (resp. ˜ F -1 / 2 ) on both sides of the first (resp. second) equation above, we arrive at

Using the definition of ̂ u t , ̂ v t +1 in (E.28), we rewrite the above as

To prove the first statement on e t 1 , the strategy is to write the LHS of (E.37) in terms of the state evolution parameters and prove that the latter quantities converge. We start with

We have

Using the spectral decomposition

we can write the first term in (E.40) as

where the first equality is by Proposition G.3 and the joint distribution of W U,t , W U,t -1 in (E.2), and the second one holds since the state evolution parameters stay at the fixed point upon initialization. So it remains to verify that Φ t,t -1 → σ ∗ 2 as t → ∞ . To see this, note that according to the state evolution recursion (E.3),

Eliminating Ψ t,t -1 from the first equation, we arrive at

whose only fixed point is σ ∗ 2 by the relations in (E.11). This concludes the proof of the first equality in (E.37). The proof of the second equality is analogous and, hence, omitted.

By using (E.37) and the fact that b → b ∗ , c → c ∗ as n →∞ (see (E.17)), we obtain

Note that the operator norm of A ∗ is almost surely bounded uniformly in n by Weyl's inequality, submultiplicativity of matrix norms and the Bai-Yin law [3]. This together with the triangle inequality of the ℓ 2 -norm and (E.38) implies that

From this, it follows that the right hand side of (E.34) (upon taken the rescaled squared norm and the sequential limits) equals

We then compute the above term by taking the SVD of A ∗ . Define two spectral projectors that are orthogonal to each other:

For the second term in (E.40), we have

where penultimate line follows since

From Lemma E.3 and the assumption that σ ∗ 2 < 1 , we know

This implies:

where the last equality holds since the limit in the first parentheses is finite by (E.36). Now (E.40) to (E.42) jointly show

Combining this with (E.36) brings us to the following identity:

which necessarily implies

as desired.

With Lemma E.4, we can complete the proof of Theorem 5.1.

Proof of Theorem 5.1. The characterization (5.7) of the top two singular values have been obtained in Lemmas E.3 and E.4. It remains to compute the overlaps which can be done using Lemma E.4 and

the state evolution (Proposition E.1). Recall the estimators ̂ u, ̂ v in (5.4) and their heuristic derivation in (E.23). Then

establishing the first equality in (5.8). The second equality in (5.8) and other quantities in (5.9) and (5.10) can be similarly obtained. The proof is completed.

F Proof of Proposition E.1

Recall ˜ u ∗ , ˜ v ∗ from (C.7) and let

F.1 Auxiliary AMP and its state evolution

For (˘ g t : R n → R n , ˘ f t +1 : R d → R d ) t ≥ 0 , the iterates of the auxiliary AMP, initialized at ˚ u -1 = 0 d and some ˚ v 0 ∈ R d , are updated according to the following rules for every t ≥ 0 :

The state evolution result associated with the above auxiliary AMP iteration asserts that the distributions of (˘ u 0 , ˘ u 1 , · · · , ˘ u t ) , (˘ v 1 , ˘ v 2 , · · · , ˘ v t +1 ) converge (in the sense of (F.16)) respectively to the laws of the random vectors ( ˘ U 0 , ˘ U 1 , · · · , ˘ U t ) , ( ˘ V 1 , ˘ V 2 , · · · ˘ V t +1 ) defined below:

where

The parameters ˘ µ t , ˘ ν t +1 ∈ R , ˘ Φ t = ( ˘ Φ r,s ) 0 ≤ r,s ≤ t , ˘ Ψ t +1 = ( ˘ Ψ r +1 ,s +1 ) 0 ≤ r,s ≤ t ∈ R ( t +1) × ( t +1) are defined recursively through the following state evolution equations:

˜

where ˘ f 0 is determined by ˚ v 0 through Assumption (A4) below. In particular,

We require the following assumptions to guarantee the existence and finiteness of the state evolution parameters defined above.

(A4) ˚ v 0 is independent of ˜ W but may depend on ˜ v ∗ . Assume that

exists and is finite. There exists a uniformly pseudo-Lipschitz function ˘ f 0 : R d → R d of order 1 such that

and for every uniformly pseudo-Lipschitz function ϕ : R d → R d of finite order, the following two limits exist, are finite and equal:

Let ˜ ν ∈ R , ˜ τ ∈ R ≥ 0 . For any s ≥ 1 ,

exists and is finite, where W V ∼ N (0 d , I d ) is independent of ˜ V ∗ .

(A5) Let ˜ ν ∈ R , and T ∈ R 2 × 2 be positive definite. For any r, s ≥ 1 ,

′ ∗

exist and are finite, where ( N,N ) ∼ N (0 2 d , T ⊗ I d ) is independent of ˜ v . Let µ ∈ R , and S ∈ R 2 × 2 be positive definite. For any r, s ≥ 0 ,

exist and are finite, where ( M,M ′ ) ∼ N (0 2 n , S ⊗ I n ) is independent of ˜ U ∗ .

Proposition F.1 (State evolution for auxiliary AMP (F.2)) . For every t ≥ 0 , let (˘ g t : R n → R n ) n ≥ 1 and ( ˘ f t +1 : R d → R d ) d ≥ 1 be uniformly pseudo-Lipschitz of finite order subject to Assumption (A5) . Consider the auxiliary AMP iteration in (F.2) defined by (˘ g t , ˘ f t +1 ) t ≥ 0 and initialized at ˚ u -1 = 0 n and some ˚ v 0 ∈ R d subject to Assumption (A4) . For any fixed t ≥ 0 , let ( ϕ : R ( t +2) n → R ) n ≥ 1 and ( ψ : R ( t +2) d → R ) n ≥ 1 be uniformly pseudo-Lipschitz functions of finite order. Then,

where ( ˘ U s , ˘ V s +1 ) 0 ≤ s ≤ t are defined in (F.3) .

Proof of Proposition F.1. By definitions of the auxiliary AMP (F.2) and the matrix ˜ A in (F.1), we have that for every t ≥ 0 ,

For every t ≥ 0 , let us consider a pair of related iterates p t , q t +1 with initialization

and update rules:

where ˘ µ t , ˘ ν t +1 are as in (F.6) to (F.8).

Informally, the above iterates are related to ˘ u t , ˘ v t +1 via

where the 'equalities' hold only in the large n limit. These relations will be made formal in the rest of the proof.

The algorithm (F.18) takes the form of a standard AMP iteration with non-separable denoising functions as in [13, 34] for which the following state evolution result applies. For any t ≥ 0 and uniformly pseudo-Lipschitz functions ϕ, ψ of finite order, it holds that

where ( ˜ U ∗ , ˘ σ 0 ˘ W U, 0 , · · · , ˘ σ t ˘ W U,t ) and ( ˜ V ∗ , ˘ τ 1 ˘ W V, 1 , · · · , ˘ τ t +1 ˘ W V,t +1 ) are defined in (F.4) and (F.5). Note that [34] allows additional randomness independent of ˜ W that goes into the denoising functions. So the asymptotic guarantee in (F.20) holds for the joint tuple involving ˜ U ∗ , ˜ V ∗ as well. (F.20) immediately implies

where we recall the definition of ˘ U t , ˘ V t +1 in (F.3). We will show that

which, when combined with (F.21), concludes the proof of Proposition F.1.

To show (F.22), suppose that ϕ, ψ are uniformly L -pseudo-Lipschitz of order k . Then by the triangle inequality,

for some L ′ depending only on t, k, L . Similar manipulation gives

Clearly, (F.22) holds if for every t ≥ 0 ,

˘

ν

t

which, together with the following statements

+1

<

∞

,

will be shown in the sequel by induction on t ≥ 0 .

Base case. Consider t = 0 . From (F.21),

where the last equality is by (F.3). Due to (F.13) and Assumption (A4), both ˘ µ 0 and ˘ σ 0 are finite, so (F.29) holds for t = 0 . Consequently, (F.23) also holds for t = 0 . Since by (F.17),

therefore (F.25) for t = 0 follows from (F.6) and Assumption (A4). This in turn implies, when combined with the finiteness of (F.31), that

˘

τ

t

+1

<

∞

,

(F.30)

verifying (F.24) for t = 0 . Since ˘ g 0 is uniformly pseudo-Lipschitz of finite order, so is the function 1 n ∑ n i =1 ∂ i ˘ g 0 ( · ) i . (F.23) to (F.25) (for t = 0 ) together imply

Using the the pseudo-Lipschitzness of ˘ g 0 again,

The last equality holds because of (F.25) (for t = 0 ) and the finiteness of (F.31) and (F.32). To show (F.28) for t = 0 , we use (F.2), (F.17) and (F.18) to write

By (F.34) and the Bai-Yin law [3],

Using (F.34) again,

where in the second line, the first equality is by (F.22) and the pseudo-Lipschitzness of ˘ g 0 , and the second equality is by the definition (F.8). We further show the finiteness of ˘ ν 1 . Note that

The first term can be bounded as

where the last step is the elementary inequality ( a + b ) 2 ≤ 2( a 2 + b 2 ) . The RHS above is finite since

whose all moments are finite by finiteness of ˘ µ 0 , ˘ σ 0 . This shows the first bound in (F.30) for t = 0 . Recalling (F.36), we then have

Therefore, (F.35), (F.39) and (F.40) altogether verify (F.28) for t = 0 .

We then show (F.26) for t = 0 . Since ˘ ν 1 < ∞ , it suffices to only consider q 1 . According to (F.17) and (F.18),

Pseudo-Lipschitzness of ˘ g 0 , finiteness (F.23) (for t = 0 ) and finiteness (F.33) jointly imply

from which (F.26) follows. This combined with (F.28) (for t = 0 ) also implies (F.27) (for t = 0 ). Finally, we are left with the second inequality in (F.30). From the definition (F.15),

which has already been shown to be finite; see (F.37). So the base case is finished.

Induction step. Assume that (F.23) to (F.28) all hold up to the t -th step (for an arbitrary t ≥ 1 ). We now show that they hold for t +1 . The idea is similar to the base case. We briefly lay down the key steps for (F.23) to (F.25) and (F.29), and omit the verification of (F.26) to (F.28) and (F.30). Using (F.3) and (F.21),

Using the definition (F.14) and the pseudo-Lipschitzness of ˘ f t +1 ,

for some L ′ depending only on k, L . The inequality is obtained in a similar way to (F.37). By induction hypothesis (F.30) and the compactness of supp(Σ) , all moments of

are finite. Therefore ˘ σ 2 t +1 < ∞ , giving the second bound in (F.29). Similarly, using the definition (F.7) and Cauchy-Schwarz,

which is again finite for the same reason as ˘ σ t +1 , giving the first bound in (F.29). Therefore (F.41) is also finite, verifying (F.23) for t +1 .

We then show (F.25) for t +1 . Using the recursions (F.2) and (F.18),

Consider T ′ 1 . Since (F.26) to (F.28) are assumed to hold, by pseudo-Lipschitzness of ˘ f t +1 ,

This with the Bai-Yin law [3] gives

Consider T ′ 2 . Recall that ˘ µ t +1 < ∞ . Using (F.7), (F.21) and (F.42) and following the argument leading to (F.36), we have

Consider T ′ 3 . By the triangle inequality,

Since (F.26) to (F.28) are assumed to hold, by pseudo-Lipschitzness of 1 n ∑ d i =1 ∂ i ˘ f t +1 ( · ) i ,

Similarly, pseudo-Lipschitzness of ˘ g t and the hypothesis (F.23) to (F.25) ensures

So combining (F.45) and (F.46), we have

(F.43), (F.44) and (F.47) altogether verify (F.25) for t +1 , and therefore also (F.24) by (F.23).

The verification of (F.26) to (F.28) and (F.30) for t +1 is completely analogous and we do not repeat similar arguments. The proof is finally completed.

F.2 Proof of Proposition E.1

We will prove Proposition E.1 by reducing the AMP iteration (E.1) (and its associated state evolution (E.2) to (E.6)) to the auxiliary AMP (F.2) (and its associated state evolution (F.3) to (F.12), (F.14) and (F.15)).

Under the following change of variables

(F.2) becomes

where b t +1 , c t are equal to ˘ b t +1 , ˘ c t , respectively, but are expressed using the derivatives of f t +1 , g t . Specifically,

The second equality follows since by (F.49),

The third equality is by the chain rule for derivatives (Proposition G.6). A similar computation gives

We now see that under the change of variables (F.48) and (F.49), the AMP iteration (E.1) can be cast as (F.2). Therefore, applying the same change of variables to the state evolution of (F.2) will produce the state evolution of (E.1). We describe the required modifications below.

The state evolution result in Proposition F.1 for the AMP in (F.2) says that the iterates v ∗ , ˘ v 1 , ˘ v 2 , · · · , ˘ v t +1 ∈ R d and u ∗ , ˘ u 0 , ˘ u 1 , · · · , ˘ u t ∈ R n converge (in the sense of (F.16a) and (F.16b)) respectively to V ∗ , ˘ V 1 , ˘ V 2 , · · · , ˘ V t +1 ∈ R d and U ∗ , ˘ U 1 , ˘ U 2 , · · · , ˘ U t ∈ R n . Recall that AMPs (E.1) and (F.2) operate on the following matrices respectively:

In view of (F.48), to obtain the analogous state evolution result for the AMP in (E.1), the definition (F.3) of ˘ U t , ˘ V t +1 should be multiplied by Ξ -1 / 2 , Σ -1 / 2 respectively. This gives the new definition of U t , V t +1 in (E.6). By the relation (F.49), the parameters ˘ µ t , ˘ ν t +1 in ˘ U t , ˘ V t +1 should be modified as follows: replace ˘ f t ( ˘ V t ) , ˘ g t ( ˘ U t ) in the recursive equations (F.6) to (F.8) with Σ -1 / 2 f t ( V t ) , Ξ -1 / 2 g t ( U t ) . This gives the new definition of µ t , ν t in (E.5). Similar operations map equations (F.9) to (F.12), (F.14) and (F.15) to equations (E.3) and (E.4). Finally, under the new definition of U t , V t +1 , the convergence result (F.16a) and (F.16b) translates to (E.8a) and (E.8b), which completes the proof.

G Auxiliary lemmas

Proposition G.1 (Gaussian integral) . Let A ∈ R d × d be a positive-definite matrix and b ∈ R d . Then

Proposition G.2. Let V ∼ Q ⊗ d where Q is a fixed distribution on R with mean 0 . Let B ∈ R d × d denote a sequence (indexed by d ) of deterministic matrices such that the empirical spectral distribution of 1 d B converges to the law of a random variable B . Then

Let B ∈ R d × d denote a sequence (indexed by d ) of deterministic matrices such that the empirical spectral distribution of 1 d B converges to the law of a random variable B . Then

Proposition G.4 (Nishimori identity) . Let ( X,Y ) be two random variables. Let k ≥ 1 and X 1 , · · · , X k be k i.i.d. samples (given Y ) from the distribution law( X | Y ) . Denote by ⟨·⟩ , E [ · ] the expectations with respect to law( X | Y ) and law( X,Y ) , respectively. Then for all continuous bounded function f , it holds that

Proposition G.5 (Conditional distribution of Gaussians) . Let d ≥ 2 and 1 ≤ p ≤ d -1 be integers. Let

be a d -dimensional Gaussian random vector, where G 1 ∈ R p , G 2 ∈ R d -p , µ 1 ∈ R p , µ 2 ∈ R d -p , Σ 1 , 1 ∈ R p × p , Σ 1 , 2 ∈ R p × ( d -p ) , Σ 2 , 2 ∈ R ( d -p ) × ( d -p ) . Then for any g 2 ∈ R d -p , the distribution of G 1 conditioned on G 2 = g 2 is given by G 1 | { G 2 = g 2 } ∼ N ( µ ′ 1 , Σ ′ 1 ) where

and Σ -1 2 , 2 denotes the generalized inverse of Σ 2 , 2 .

where V ∼ Q .

Proposition G.3. Let

Proposition G.6 (Chain rule of derivatives) . Let A ∈ R n × n and f : R n → R n . Let x ∈ R n and ˜ x := Ax . Then for any i, j ∈ [ n ] ,

where f ( ˜ x ) j ∈ R denotes the j -th ( j ∈ [ n ] ) output of f ( ˜ x ) ∈ R n .

Proof. The proof follows from elementary applications of the chain rule for derivatives. Writing A = [ a 1 · · · a n ] ⊤ and f = [ f 1 · · · f n ] ⊤ , we have

where ∂ k f j denotes the partial derivative of f j : R n → R with respect to its k -th argument. Then,

as claimed.

Proposition G.7 (Stein's lemma [68]) . Let W ∼ N (0 , σ 2 ) and let f : R → R be such that both expectations below exist. Then E [ Wf ( W )] = σ 2 E [ f ′ ( W )] .

Proposition G.8 (Gaussian Poincaré inequality [16, Theorem 3.20]) . Let X ∼ N (0 n , I n ) and f : R n → R a differentiable function. Then

Proposition G.9 (Bounded difference inequality [16, Corollary 3.2]) . Let U ⊂ R and f : U n → R a function such that there exist c = ( c 1 , · · · , c n ) ∈ R n ≥ 0 satisfying for all i ∈ [ n ] ,

Then if X ∈ U n is a random vector consisting of independent elements, we have Var[ f ( X )] ≤ ∥ c ∥ 2 2 / 4 .

Proposition G.10 ([62, Lemma 3.2]) . If f, g are differentiable convex functions, then for any a ∈ R and a ′ > 0 ,

where

Definition G.1 (Monotone conjugate) . Let f : R ≥ 0 → R be a non-decreasing convex function. Its monotone conjugate f ∗ is defined as

Proposition G.11 ([53, Proposition C.1]) . Let I ⊂ R be an interval and ( f n : I → R ) n be a sequence of convex functions converging pointwise to f . Then for all t ∈ I such that the following quantities exist,

Proposition G.12 ([53, Proposition C.6]) . Let f, g : R ≥ 0 → R be strictly convex differentiable functions and

Then

and sup ( q 1 ,q 2 ) ∈C and sup q 1 ,q 2 ≥ 0 are achieved at the same ( q ∗ 1 , q ∗ 2 ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims match theoretical and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Global assumptions are given in Section 3 and all theoretical results are formally stated and proved.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: All experiments use synthetic data and can be reproduced given the instructions in Section 5. Data and code are not released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Figures 1 to 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: All experiments are synthetic and can be run efficiently on standard personal computers.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and confirm that this research conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This research is purely theoretical and has no obvious societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This research is purely theoretical and has no risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the following \"matrix denoising\"/low-rank estimation problem. Given a rectangular matrix $Y = uv^\\top + W$, the goal is to recover the rank-one factors $u,v$, when $W$ is random, with as little $\\ell_2$ error as possible.\n\nA classic and well-studied setting takes $W$ to have iid entries; often one even assumes that the entries are Gaussian. This setting has been studied in random matrix theory, high-dimensional statistics, and more recently via techniques from statistical physics.\n\nThe iid noise assumption is quite strong, and so a more recent line of works aims to relax that assumption by allowing correlations among the entries of $W$. In the \"two-sided\" heteroscedastic noise model, $W$ takes the form $\\Theta^{1/2} \\cdot W' \\cdot \\Sigma^{1/2}$, where $\\Theta$ and $\\Sigma$ are PSD matrices and $W'$ has iid Gaussian entries. The assumption is that $\\Theta$ and $\\Sigma$ are known.\n\nThe main contributions of this paper are twofold:\n\n- a new spectral estimator whose performance improves over more naive spectral methods for recovery of $u$ and $v$.\n- a proof that under slightly stronger assumptions ($u$,$v$ Gaussian), the spectral estimator obtains nontrivial $\\ell_2$ error whenever this is information-theoretically possible, and even obtains information-theoretically optimal $\\ell_2$ error when the heteroscedasticity is one-sided.\n\nAlong the way, the paper establishes a simple formula for the information-theoretic signal-to-noise threshold governing when nontrivial recovery of $u$ and $v$ is possible in this model. The paper also performs numerical experiments on synthetic data to validate the theory.\n\nSTRENGTHS:\n- Well-written and easy to read first 9 pages\n- Thorough mathematical investigation of the two-sided heteroscedastic spiked matrix model\n- New algorithm with nice optimality guarantees\n\nWEAKNESSES:\nMy main reservation is that, as with a lot of papers using stat phys techniques, one gets the feeling that the assumptions are sort of designed to make the mathematical techniques work. I think the biggest offender here is the assumption that $\\Sigma$ and $\\Theta$ are known. Less major are the Gaussian-ness assumptions, the assumption that $n,d$ are of comparable order, and the assumption that the empirical spectral distributions of $\\Theta$ and $\\Sigma$ converge -- I think these allow the use of asymptotic methods, basically.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study how to recover a rank one spike corrupted by doubly heteroscedastic gaussian noise in the high dimensional regime. We are given a condition on the signal to noise ratio to indicate whether it's information-theoretically possible to have a non-trivial recovery of the spike. If this is satisfied then there is spectral estimator which can obtain non-trivial recovery. In particular cases this estimator is also Bayes optimal.\n\nSTRENGTHS:\nThe paper is well written and does a really good job at introducing the main ideas in an intuitive way. While this kind of spectral estimators are well known in the physics literature, their application to such a noise model is an interesting generalisation. \n\nI believe this paper thoroughly explores this denoising problem, with the only easily achievable extension being looking at a rank $r$ spike in the signal (where r is a constant in d,n), which I am sure however wouldn't alter the results significantly. Thus this is in my opinion quite a solid contribution with not much room for improvement.\n\nWEAKNESSES:\nThe paper introduces no significant advances in the theoretical tools or understanding of matrix denoising as all the tools used are essentially well known. I believe one issue in the writing is its lack of clarity in stating which results are completely rigorous and which aren't. The authors are upfront in saying they are guided by physics-inspired heuristics, but reading in the appendix it seems like (at least in some sections) the derivations are quite solid. A tangible improvement to the writing would be to state explicitly which results are conjectures and which are theorems.\n\nI think it would greatly improve the readability of figures 2 and 3 to have the error on the mean instead of the std.\n\nI would add a sentence describing more explicitly the difference and respective advantages of (3.1) (where the spectrum is O(1)) and (4.1) (where the elements of Y are O(1)).\n\nSmall typos:\n\n1. Line 203: $\\sigma_2$ is not directly defined. You will only do so in Theorem 5.1\n2. Line 188: you invoke the Nishimori identity but don't state it in the main text\n3. Line 200: you say that $\\eta>0$. While I also expect this to be true, I think you mean to say that they are real. The same applies to all the square roots in 5.3 and 5.4\n\n\nFinally, not exactly a typo but I personally find it confusing to use $\\bar \\Sigma$ and $\\bar \\Xi$ when one wants to average over the singular values of $\\Sigma$ and $\\Xi$. I would prefer having an explicit integral over the singular value PDF.\n\nQUESTIONS:\nWould it be possible to look at the singular values of $A$ before and after the pre-processing? Can we clearly see a spike emerging if 5.1 is true by doing the pre-processing?\n\nYou state that AMP has the fundamental limitation that requires a \"warm start\" to be effective. While this is true, initialising the estimator at random from the prior should allow it to have non-zero overlap in O(log(d)) steps. Do you see this in your numerics?\n\nCould you run AMP after using the spectral estimator and put the additional lines and simulation dots in figures 2, 3?\n\nLIMITATIONS:\nThe limitations are correctly addressed in section 6. There is no negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper considers the problem of matrix denoising. Given an observation X = A + W where W is noise, our goal is to estimate A, which is typically low-rank. Unlike previous works, this paper treats the case where W is doubly heteroscedastic. The authors identify a condition for non-trivial estimation of the signal vectors, along with an accompanying spectral algorithm that succeeds whenever that condition holds, under a technical condition.

STRENGTHS:
The paper addresses an important problem using a novel approach using statistical physics/ AMP concepts. The results, both theoretical and empirical, are strong and add considerably to the literature.

WEAKNESSES:
Strictly speaking, you do not show that whitening fails, only that the whitened matrix does not match the proposed AMP approach (lines 262-265).

QUESTIONS:
Initially you say that you believe (5.1) implies \sigma_2^{\star} < 1, and later you say that you believe these conditions are equivalent. Could you please clarify which one it is?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13