Title: Optimal Algorithms for the Inhomogeneous Spiked Wigner Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spectral Method

Community detection

Wigner Spike model

Random Matrix

BBP transition

Approximate Message Passing

Spin glasses

Statistical Physics
Abstract: We study a spiked Wigner problem with an inhomogeneous noise profile. Our aim in this problem is to recover the signal passed through an inhomogeneous low-rank matrix channel. While the information-theoretic performances are well-known, we focus on the algorithmic problem. First, we derive an approximate message-passing algorithm (AMP) for the inhomogeneous problem and show that its rigorous state evolution coincides with the information-theoretic optimal Bayes fixed-point equations. Second, we deduce a simple and efficient spectral method that outperforms PCA and is shown to match the information-theoretic transition.

Optimal Algorithms for the Inhomogeneous Spiked Wigner model

Abstract

We study a spiked Wigner problem with an inhomogeneous noise profile. Our aim in this problem is to recover the signal passed through an inhomogeneous low-rank matrix channel. While the information-theoretic performances are wellknown, we focus on the algorithmic problem. First, we derive an approximate message-passing algorithm (AMP) for the inhomogeneous problem and show that its rigorous state evolution coincides with the fixed-point equations satisfied by the Bayes-optimal estimator. Second, we deduce a simple and efficient spectral method that outperforms PCA and is shown to match the information-theoretic transition.

Low-rank information extraction from a noisy data matrix is a crucial statistical challenge. The spiked random matrix models have recently gained extensive interest in the fields of statistics, probability, and machine learning, serving as a valuable platform for exploring this issue Donoho and Johnstone [1995], Péché [2014], Baik et al. [2005]. A prominent example is the spiked Wigner model, where a rank one matrix is observed through a component-wise homogeneous noise.

Here we consider the inhomogeneous version of the model which has been recently introduced in a series of papers Barbier and Reeves [2020], Behne and Reeves [2022], Alberici et al. [2021a, 2022] and naturally arises in different contexts such as community detection Behne and Reeves [2022], deep Boltzmann machines Alberici et al. [2021b] etc. In particular, it corresponds to the dense limit of the degree-corrected stochastic block model Karrer and Newman [2011] (see the explicit mapping in Guionnet et al. [2022]). In this model the signal is observed through an inhomogeneous Gaussian noise with block-constant variance profile. Such block-constant noise naturally arises in settings such as community detection and stochastic-block models. The assumption for the noise to be Gaussian is in no way restrictive as a universality result in Guionnet et al. [2022] shows that an entrywise transformation of data generated from a general inhomogeneous inference problem reduces any probabilistic noise to the spiked Wigner problem with Gaussian data. This generality is what makes the model appealing.

We now define the model: Consider a partition { 1 , . . . , N } =[ N ] into q disjoint groups C N 1 ∪ · · · ∪ C N q = [ N ] . This partition is encoded by a function g : [ N ] ↦→ [ q ] which maps each index i ∈ [ N ] into its group g ( i ) ∈ [ q ] . Let ˜ ∆ ∈ R q × q be a symmetric matrix encoding a block-constant symmetric matrix ∆ ∈ R N × N

We assume both the noise profile ˜ ∆ and the partition function g to be known. This is often the case in practical applications, such as the degree-corrected block model, Karrer and Newman [2011]

Additionally, in practical applications where the noise profile is not known, one can often empirically estimate the noise profile and assign group membership according to the empirical variances. Such an approach is likely to yield good results but lies outside the scope of the present work.

We observe the signal x ⋆ ∈ R N which is assumed to have independent identically distributed coordinates generated from some prior distribution P 0 (i.e. P ( x ⋆ = x ) = ∏ N i =1 P 0 ( x ⋆ i = x i ) ) through noisy measurements:

Here and throughout the article ⊙ denotes the Hadamard product, ∆ is the Hadamard square-root of ∆ and A is a real-valued symmetric GOE matrix with off-diagonal elements of unit variance. The Bayes-optimal performance of this model in the asymptotic limit N →∞ was studied rigorously in Guionnet et al. [2022], Behne and Reeves [2022], Chen and Xia [2022], Chen et al. [2021] who characterized the fundamental information-theoretic limit of reconstruction in this model.

√

Here we focus instead on the algorithmic problem of reconstructing the (hidden) spike. Our contributions are many-fold :

We show how one can construct an Approximate Message Passing (AMP) algorithm for the inhomogeneous Wigner problem, whose asymptotic performance can be tracked by a rigorous state evolution, generalizing the homogeneous version of the algorithm for low-rank factorization [Bayati and Montanari, 2011, Deshpande et al., 2015, Lesieur et al., 2017].

We derive a fixed-point equation for AMP and show that it coincides with the fixed-point equation for the Bayes-optimal estimator obtained in Guionnet et al. [2022]

Finally, we present a linear version of AMP [Maillard et al., 2022], that is equivalent to a spectral method. We conjecture it to be optimal in the sense that it can detect the presence of the spike in the same region as AMP. This is quite remarkable since, as shown in [Guionnet et al., 2022, Section 2.4], the standard spectral method (PCA) fails to do so.

Related work The class of approximate message passing algorithms (AMP) has attracted a lot of attention in the high-dimensional statistics and machine learning community, see e.g. [Donoho et al., 2009, Bayati and Montanari, 2011, Rangan, 2011, Deshpande et al., 2015, Lesieur et al., 2017, Gerbelot and Berthier, 2021, Feng et al., 2022]. The ideas behind this algorithm have roots in physics of spin glasses Mézard et al. [1987], Bolthausen [2014], Zdeborová and Krzakala [2016]. AMP algorithms are optimal among first order methods [Celentano et al., 2020], thus their reconstruction threshold provides a bound on the algorithmic complexity in our model. Our approach to the inhomogeneous version of AMP relies on several refinements of AMP methods to handle the full complexity of the problem, notably the spatial coupling technique Krzakala et al. [2012], Donoho et al. [2013], Javanmard and Montanari [2013], Gerbelot and Berthier [2021], Rossetti and Reeves [2023].

AMP

1.0

optimal

60

0.8

E

0.7

0.6

0.5

0.4

0.5

10

1.5

2.0

2.5

3.0

SNR

Factorizing low-rank matrices is a ubiquitous problem with many applications in machine learning and statistics, ranging from sparse PCA to community detection and sub-matrix localization. Many variants of the homogeneous problem have been studied in the high-dimensional limit [Deshpande et al., 2015, Lesieur et al., 2017, Barbier et al., 2018, Lesieur et al., 2017, Alaoui et al., 2020, Lelarge and Miolane, 2019, Louart and Couillet, 2018, Barbier and Reeves, 2020]. The inhomogeneous version was discussed in detail in Guionnet et al. [2022], Behne and Reeves [2022], Alberici et al. [2021a]. Spectral methods are a very popular tool to solve rank-factorization problems [Donoho and Johnstone, 1995, Péché, 2014, Baik et al., 2005]. Using AMP as an inspiration for deriving

new spectral methods was discussed, for instance, in Saade et al. [2014], Lesieur et al. [2017], Aubin et al. [2019], Mondelli and Montanari [2018], Mondelli et al. [2022], Maillard et al. [2022], Venkataramanan et al. [2022].

1 Main results

Message passing algorithm For each t ≥ 0 , let ( f a t ) a ∈ [ q ] be a collection of Lipschitz functions from R → R , and define for t ∈ N f t : R N ↦→ R N by

These functions are often called denoiser functions and can be chosen amongst several options, such as the Bayes optimal denoisers for practical applications (see Section 2), or even linear denoisers (see Section 3). We shall consider the following AMP recursion

with the so-called Onsager term b t = 1 N ∆ f ′ t ( x t ) ∈ R N where 1 ∆ is the Hadamard inverse of ∆ and f ′ t is the vector of coordinate wise derivatives.

In practical implementations, we initialize the algorithm with some non-null x 0 and let it run for a certain number of iterations. One efficient way to do this is the spectral initialization [Mondelli and Venkataramanan, 2021] with the method described in sec. 3. In Figure 1 we provide an example of the performance of the AMP together with the Bayes-optimal estimator predicted by the asymptotic theory. Even at very moderate sizes, the agreement between theory and simulation is clear.

State evolution AMP was the basis of many works for the homogeneous case e.g. Deshpande et al. [2015], Lesieur et al. [2017]. Our first contribution is the introduction of an AMP and of its rigorous state evolution [Javanmard and Montanari, 2013, Theorem 1] in the inhomogeneous setting. To state a well-defined limit of the AMP, we have the following assumptions.

Assumption 1.1. To ensure that our inhomogeneous AMP has a well-defined limit, we assume that

For each a ∈ [ q ] , we have

For each t ∈ [ N ] and a ∈ [ q ] , both ( f a t ) and ( f a t ) ′ are Lipschitz.

For each a ∈ [ q ] , there exists ( σ 0 a ) 2 ∈ R such that, in probability,

Our first result describes the distribution of the iterates in the limit. Our mode of convergence will be with respect to L -pseudo-Lipschitz test functions ϕ : R M → R satisfying

We define the following state evolution parameters µ t b and σ t b for b ∈ [ q ] through the recursion

where x ⋆ 0 and Z are independent. We prove that the iterates x t i are asymptotically equal in distribution to µ t g ( i ) x ⋆ 0 + σ t g ( i ) Z where x ⋆ 0 ∼ P 0 and Z is an independent standard Gaussian.

Theorem 1.2 (State evolution of AMP iterates in the inhomogeneous setting) . Suppose that Assumption 1.1 holds, and that P 0 has bounded second moment. Let ϕ : R 2 → R be a 2-pseudo-Lipschitz test functions satisfying (4) . For any a ∈ [ q ] , the following limit holds almost surely

where Z is a standard Gaussian independent from all other variables.

Remark 1.3. The notion of convergence under the L -pseudo-Lipschitz test functions induces a topology that is equivalent to the one generated by the L -Wasserstein topology [Feng et al., 2022, Remark 7.18]. We can strengthen the second moment assumption on P 0 to finite k th moment, but the induced topology will then change to the k -Wasserstein topology, see [Javanmard and Montanari, 2013, Theorem 1].

Even though the theoretical result above applies in the high-dimensional limit, numerical simulations show that even for medium-sized N (around 500 ), the behaviour of the iterates is well described by the state evolution parameters. Through the state evolution equations (5) we are able to track the iterates of the AMP iteration with just two vectors of parameters obeying the state evolution recursion: the overlap with the true signal ( µ t a ) a ∈ [ q ] and its variance ( σ t a ) a ∈ [ q ] . We next obtain the following necessary and sufficient condition for the overlaps of a fixed point of the iteration (2):

Theorem 1.4 (Bayes-Optimal fixed point) . Assume AMP satisfies Assumption 1.1 and let the denoising functions be the Bayes ones (32) . Then the overlaps µ = ( µ a ) a ∈ [ q ] in (5) satisfy the following fixed point equation

Remark 1.5. The state evolution fixed point equation above coincides with the fixed point equation satisfied by the Bayes optimal estimator in [Guionnet et al., 2022, Equation 2.14].

A spectral method Given the matrix Y defined in (2) we consider the transformed matrix

Let c = ( c a ) a ∈ [ q ] . We define the inhomogeneous signal-to-noise (SNR) ratio of such a model by

Conjecture 1.6. The top eigenvalue of ˜ Y separates from the bulk if and only if the signal to noise ratio λ ( ∆ ) > 1 . In particular, if ˆ x is the top eigenvector of ˜ Y , then if and only if λ ( ∆ ) < 1 we have:

The conjecture is based on heuristic arguments identifying the fixed point of AMP with a spectral method. This matches precisely the recovery transition in [Guionnet et al., 2022, Lemma 2.15 Part (b)]. In this paper, we rigorously show that with SNR( ∆ ) < 1 our proposed spectral method fails to recover the signal. We illustrate the eigenvalue BBP-like transition in Fig.2.

2 The inhomogeneous AMP algorithm

In this section, we derive the formula for the inhomogeneous AMP iteration (25). We first recall the general matrix framework of AMP from [Javanmard and Montanari, 2013]:

17.5

17.5

15.0

15.0

12.5

12.5

10.0

10.0

7.5

7.5

5.0

5.0

2.5

2.5

0.0

2.5

2.0

1.5

1.0

0.5

0.0

0.5

1.0

0.0

5

Matrix AMP In the matrix setting an AMP algorithm operates on the vector space V q,N ≡ ( R q ) N ≃ R N × q . Each element of v = ( v 1 , . . . , v N ) ∈ V q,N will be regarded as N -vector with entries v i ∈ R q .

Definition 2.1 (AMP) . A matrix AMP acting on this space is represented by ( A , F , v 0 ) , where:

A = G + G T , where G ∈ R N × N has iid entries G ij ∼ N (0 , 1 2 ) .

2. F is a family of N Lipschitz functions f i t : R q ↦→ R q , i ∈ [ N ] indexed by time t ∈ N . The family F encodes a function f t : V q,N →V q,N that acts separately on each coordinate v j ∈ R q ,

3. v 0 ∈ V q,N is a starting condition.

The algorithm itself is a sequence of iterates generated by:

where B t is the q × q Onsager matrix given by

where ∂f j t denotes the Jacobian matrix of f j t . The limiting properties of the AMP sequences are well known and can be found in [Javanmard and Montanari, 2013, Theorem 1].

The inhomogeneous AMP -Wenow define an inhomogeneous AMP iteration which takes into account the block-constant structure of the noise:

Definition 2.2. An inhomogeneous AMP on V 1 ,N = R N is represented by ( A , F , x 0 , ∆ ) , where the terms A , F , x 0 are defined in Definition 2.1 and ∆ is the N × N variance profile encoded by ˜ ∆ ∈ R q × q and grouping g : [ N ] → [ q ] defined by (1) . We further assume that the family of functions F is encoded by functions f a t : R ↦→ R for a ∈ [ q ] which define the group dependent function

The sequence of iterates x t ∈ R N of the ( A , F , v 0 , ∆ ) are defined as follows:

where 1 √ ∆ is the Hadamard inverse square root of the noise, and the Onsager term b t has the following form

State evolution of the inhomogeneous AMP Through a continuous embedding, we will reduce our inhomogeneous AMP to the matrix AMP framework, and recover the state evolution of the inhomogeneous AMP. We define the diagonal matrix operator blockdiag : R N ↦→ V q,N which outputs a block diagonal matrix according to the block structure of our discretization of [ N ] :

Likewise, we define the projection operator blockproj : V q,N ↦→ R N which extracts a vector of size N from a N × q according to the block structure of [ N ] by

Under these changes of variables, we define

and ˜ f t : ( R q ) N ↦→ ( R q ) N

We encode the family of functions ˜ f t by ˜ F ( ∆ ) .

Lemma 2.3. Let x t be iterates from the AMP ( A , F , v 0 , ∆ ) . Then the iterates r t := blockdiag( x t ) follow the generalized matrix AMP ( A , ˜ F ( ∆ ) , r 0 ) .

Proof. We will show that the projection of the iterates r t from ( A , ˜ F ( ∆ ) , r 0 ) are the iterates from ( A , F , v 0 , ∆ ) . It is easy to check that

Next, notice that Jacobian is given by

which is a matrix where only the column number g ( i ) has non-zero elements. Applying (11), we thus get for a, b ∈ [ q ] × [ q ]

It follows that

As a consequence, the inhomogeneous state evolution equations in Theorem 1.2 follow immediately from the state evolution equations of ( A , ˜ F ( ∆ ) , r 0 ) discussed in [Javanmard and Montanari, 2013, Section 2.1]. This follows from the observation that given the law of r t in the high dimensional limit, the law of x t = blockproj( r t ) is straightforward to compute. We define

We will show that the distribution of the iterate x t i is asymptotically normal with mean 0 and variance ( σ t g ( i ) ) 2 .

Lemma 2.4 (Behavior of the inhomogeneous AMP iterates) . Suppose that Assumption 1.1 holds, and that P 0 has a bounded second moment. Let ϕ : R 2 → R be a L -pseudo-Lipschitz test function satisfying (4) . For any a ∈ [ q ] , then the following limit holds almost surely

where Z is an independent standard Gaussian.

Proof. In matrix-AMP [Javanmard and Montanari, 2013, Th. 1], the marginals of the iterates r t from ( A , ˜ F ( ∆ ) , r 0 ) are approximately Gaussian and encoded by the positive definite matrices

We now show that a ∈ [ q ] and i ∈ C N a , ̂ Σ t a depends only on ( σ t a ) 2 = Σ t aa . Indeed, by the definition of ˜ f i t we have

where Z t a ∼ N (0 , ( σ t a ) 2 ) is the a th component of the Gaussian vector Z t . The key observation here is that by construction our function ˜ f i t , R q ↦→ R q depends only on the ith component Z t i of the Gaussian vector Z t . To characterize the limiting distribution of x t = blockproj( r t ) , we only need to keep track of the variances ( σ t j ) 2 , j ∈ [ N ] . Using (21), for a given a ∈ [ q ] and i ∈ C N a we write

Finally, with (20) we get that for any b ∈ [ q ] and any j ∈ C N b , using Z t a ∼ N (0 , ( σ t a ) 2 ) ,

The inhomogeneous spiked Wigner model in the light of the AMP approach We now generalize the state evolution equations from Lemma 2.4 to spiked matrices with an inhomogenous noise profile as was stated in Theorem 1.2. This reduction via a change of variables is standard, see for example [Deshpande et al., 2015, Lemma 4.4]. Remember that in the inhomogeneous version of the spiked Wigner model we observe the signal x ⋆ through an inhomogeneous channel:

Our AMP algorithm is defined with the following recursion:

where b t = 1 N ∆ f ′ t ( x t ) and f is encoded by the family of functions in Definition 2.2. The main difference in contrast to the iteration (13) is that our data matrix Y is no longer a centered matrix, while 1 √ ∆ ⊙ A is. We would like to reduce (25) to an iteration of the form (13) with respect to a different parameter s t which is uniquely determined by x t

Doing so will allow us to recover the limiting laws of the iterates from Lemma 2.4. This is done via a standard change of variables to recenter Y . We will sketch the argument in this section, and defer the full proof of Theorem 1.2 to the Appendix A in the Supplementary Material.

To simplify notation,let us denote f t ( x t ) := ˆ x t . We proceed following the approach of [Deshpande et al., 2015, Lemma 4.4]. We rewrite (25) using the definition of Y to get

If indices are independent, then by the strong law of large numbers one would expect that

where i a is some index belonging to the group C N a and x ⋆ 0 is a random variable distributed according to the prior distribution P 0 . For b ∈ [ q ] and i ∈ C N b , we define the block overlap µ t b using the recursion

where Z is a standard Gaussian random variable independent from all others sources of randomness. Notice that (28) is precisely the asymptotic behavior of the summation appearing in (27).

We now make a change of variables and track the iterates

where µ 0 is the vector of block overlaps of the initial condition x 0 with the truth. We reduced the (25) iteration to the following iteration in which we easily recognize a version of (13):

with the initial condition s 0 = x 0 -µ 0 ⊙ x ⋆ and the Onsager term taken from (14) is given by

Using Lemma 2.4, we can recover the asymptotic behavior of the iterates x t given in (25) by computing the iterates s t + µ t ⊙ x ⋆ where s t follows (30) and µ t satisfies (28). From this reduction we obtain the following state evolution equations describing the behaviour of (25):

This (informally) characterizes the limiting distribution of the state evolution of the iterates from the inhomogeneous AMP stated in Theorem 1.2.

Fixed-point equation of state evolution in the Bayes-optimal setting Suppose that we know the prior distribution P 0 of x ⋆ 0 . The Bayes-optimal choice for the denoising functions f j t , j ∈ [ N ] is simply the expectation of x ⋆ 0 with respect to the posterior distribution,

Under this Bayes-optimal setting, we can simplify the equations obtained in the previous section and see that AMP estimator is indeed an optimal one by studying its fixed point.

Proof of Theorem 1.4. For this choice of f j t the Nishimori identity (see for example [Lelarge and Miolane, 2019, Proposition 16]) states that for a ∈ [ q ] and j ∈ C N a ,

In this setting, the state evolution equations from Theorem 1.2 reduce to

Remarkably with the Bayes-optimal choice of the denoising functions we have that for t ≥ 1 for each block b ∈ [ q ] , µ t +1 b = ( σ t +1 b ) 2 . Therefore a necessary and sufficient condition for an estimator to be a fixed point of the state evolution is to simply have its overlaps µ t b unchanged by an iteration of the state evolution. This translates into the following equation for the overlaps µ b , b ∈ [ q ]

The result of Theorem 1.4 now follows immediately.

3 A spectral method adapted to the inhomogeneous spiked Wigner model

From AMP to a spectral method Remarkably, AMP and the state evolution machinery associated with it can help us design a simple spectral algorithm that matches the informationtheoretic phase transition [Guionnet et al., 2022, Remark 2.16]. Recall that Theorem 1.2 does not require the denoising functions f t to be Bayes-optimal, but can be applied to any Lipschitz family of functions. In this section, we analyze the state evolution for the family of identity functions, f t ( x ) = x . We can gain some intuition motivating such a choice by considering simple priors, such as the Rademacher prior. In the case of the Rademacher prior a simple computation shows that the Bayes-optimal choice of the denoising functions yields f j t ( ⋆ ) = tanh ( ⋆ ) . In the first order approximation we have tanh ( x ) ≈ x . Thus, at least for the Rademacher prior, the choice of identity functions as denoising functions corresponds to the first order approximation of the Bayes-optimal choice. By Remark 3.1, we can assume that the entries of x ⋆ have unit variance. With this choice of denoising functions the AMP iteration will simply become:

If we denote B t = diag( b t ) , it is easy to see that the fixed point of this iteration yields

so any x fixed by the AMP iteration (36) must be an eigenvector of the matrix

A simple spectral method consists in taking the principal eigenvector (associated to the largest eigenvalue) of the matrix ˜ Y = Y √ N ∆ -B t .

Analysis of the spectral method using state evolution The spectral algorithm described above behaves like the AMP iteration with identity denoising functions around its fixed point. Therefore we can analyze this spectral algorithm using state evolution machinery for the AMP iteration. In the case of identity functions we have f a t ( x ) = x for all a ∈ [ q ] , so

which transforms state evolution equations (5) into the following simple form:

Rewriting the overlap state evolution in a matrix form we get for c = ( c a ) a ∈ [ q ] that

The special form of state evolution above is rather informative and we can interpret it as follows. First of all, in the regime where λ ( ∆ ) = ∥ ∥ diag( √ c ) 1 ∆ diag( √ c ) ∥ ∥ op < 1 , we can see from (42) that any iteration of the AMP recursion (36) contracts the vector overlap (multiplying it by a matrix with the operator norm smaller than 1). Thus the only possible overlap of a fixed point of (36) is 0. Moreover, we have defined the matrix ˜ Y in a way that any eigenvector of this matrix is a fixed point of the AMP recursion (36). Thus any eigenvector of ˜ Y must have zero expected overlap with the signal, meaning that the spectral method is uninformative in the regime λ ( ∆ ) < 1 .

Second of all, even though we cannot say anything rigorous in the regime λ ( ∆ ) > 1 , looking at state evolution equations of the AMP recursion (36), we see instability (overlap blows up but so does the variance). We conjecture that the principal eigenvector of ˜ Y correlates with the signal exactly in the regime λ ( ∆ ) > 1 which would imply that the BBP transition happens precisely at λ ( ∆ ) = 1 .

Remark 3.1. In general, we can let γ = E x ⋆ 0 [( x ⋆ 0 ) 2 ] and consider the normalized matrix

for ¯ x = x √ γ and ¯ ∆ = ∆ γ 2 . Notice that the entries of ¯ x now have unit variance. Under this setting, the transition of the transformation in (38) applied to ¯ Y , which appears in (7) , has transition at

which is the generalized SNR defined in (8) .

Acknowledgments and Disclosure of Funding

We thank Alice Guionnet and Lenka Zdeborova for valuable discussions. We acknowledge funding ´ from the ERC Project LDRAM: ERC-2019-ADG Project 884584, and by the Swiss National Science Foundation grant SNFS OperaGOST, 200021 200390

References

David L Donoho and Iain M Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the american statistical association , 90(432):1200-1224, 1995.

Sandrine Péché. Deformed ensembles of random matrices. In Proceedings of the International Congress of Mathematicians, Seoul , volume 3, pages 1059-1174, 2014.

Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. , 33(5):1643-1697, 2005. ISSN 0091-1798. doi: 10.1214/009117905000000233. URL http://dx.doi.org/10.1214/ 009117905000000233 .

Jean Barbier and Galen Reeves. Information-theoretic limits of a multiview low-rank symmetric spiked matrix model. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2771-2776. IEEE, 2020.

Joshua K Behne and Galen Reeves. Fundamental limits for rank-one matrix estimation with groupwise heteroskedasticity. In International Conference on Artificial Intelligence and Statistics , pages 8650-8672. PMLR, 2022.

Diego Alberici, Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. The multi-species mean-field spin-glass on the nishimori line. Journal of Statistical Physics , 182:1-20, 2021a.

Diego Alberici, Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. A statistical physics approach to a multi-channel wigner spiked model. Europhysics Letters , 136(4):48001, 2022.

Diego Alberici, Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. The solution of the deep Boltzmann machine on the Nishimori line. Comm. Math. Phys. , 387(2):1191-1214, 2021b. ISSN 0010-3616. doi: 10.1007/s00220-021-04165-0. URL https://doi.org/10.1007/ s00220-021-04165-0 .

Brian Karrer and Mark EJ Newman. Stochastic blockmodels and community structure in networks. Physical review E , 83(1):016107, 2011.

Alice Guionnet, Justin Ko, Florent Krzakala, and Lenka Zdeborová. Low-rank matrix estimation with inhomogeneous noise. arXiv preprint arxiv:2208.05918 , 2022.

Hong-Bin Chen and Jiaming Xia. Hamilton-Jacobi equations for inference of matrix tensor products. Ann. Inst. Henri Poincaré Probab. Stat. , 58(2):755-793, 2022. ISSN 0246-0203. doi: 10.1214/ 21-aihp1183. URL https://doi.org/10.1214/21-aihp1183 .

Hong-Bin Chen, Jean-Christophe Mourrat, and Jiaming Xia. Statistical inference of finite-rank tensors, 2021. URL https://arxiv.org/abs/2104.05360 .

Mohsen Bayati and Andrea Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory , 57(2):764-785, 2011. doi: 10.1109/TIT.2010.2094817.

Yash Deshpande, Emmanuel Abbe, and Andrea Montanari. Asymptotic mutual information for the two-groups stochastic block model. CoRR , abs/1507.08685, 2015. URL http://arxiv.org/ abs/1507.08685 .

Thibault Lesieur, Florent Krzakala, and Lenka Zdeborová. Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications. Journal of Statistical Mechanics: Theory and Experiment , 2017(7):073403, 2017.

Antoine Maillard, Florent Krzakala, Yue M Lu, and Lenka Zdeborová. Construction of optimal spectral methods in phase retrieval. In Mathematical and Scientific Machine Learning , pages 693-720. PMLR, 2022.

David L Donoho, Arian Maleki, and Andrea Montanari. Message-passing algorithms for compressed sensing. Proceedings of the National Academy of Sciences , 106(45):18914-18919, 2009.

Sundeep Rangan. Generalized approximate message passing for estimation with random linear mixing. In 2011 IEEE International Symposium on Information Theory Proceedings , pages 2168-2172. IEEE, 2011.

Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. arXiv preprint arXiv:2109.11905 , 2021.

Oliver Y Feng, Ramji Venkataramanan, Cynthia Rush, Richard J Samworth, et al. A unifying tutorial on approximate message passing. Foundations and Trends® in Machine Learning , 15(4):335-536, 2022.

Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications , volume 9. World Scientific Publishing Company, 1987.

Erwin Bolthausen. An iterative construction of solutions of the tap equations for the sherringtonkirkpatrick model. Communications in Mathematical Physics , 325(1):333-366, 2014.

Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics , 65(5):453-552, 2016.

Michael Celentano, Andrea Montanari, and Yuchen Wu. The estimation error of general first order methods. In Conference on Learning Theory , pages 1078-1141. PMLR, 2020.

Florent Krzakala, Marc Mézard, Francois Sausset, Yifan Sun, and Lenka Zdeborová. Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices. Journal of Statistical Mechanics: Theory and Experiment , 2012(08):P08009, 2012.

David L Donoho, Adel Javanmard, and Andrea Montanari. Information-theoretically optimal compressed sensing via spatial coupling and approximate message passing. IEEE transactions on information theory , 59(11):7434-7464, 2013.

Adel Javanmard and Andrea Montanari. State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference: A Journal of the IMA , 2(2):115-144, 2013. doi: 10.1093/imaiai/iat004.

Riccardo Rossetti and Galen Reeves. Approximate message passing for the matrix tensor product model, 2023.

Jean Barbier, Mohamad Dia, Nicolas Macris, Florent Krzakala, and Lenka Zdeborová. Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method. arXiv preprint arXiv:1812.02537 , 2018.

Ahmed El Alaoui, Florent Krzakala, and Michael Jordan. Fundamental limits of detection in the spiked Wigner model. The Annals of Statistics , 48(2):863 - 885, 2020. doi: 10.1214/19-AOS1826. URL https://doi.org/10.1214/19-AOS1826 .

Marc Lelarge and Léo Miolane. Fundamental limits of symmetric low-rank matrix estimation. Probab. Theory Related Fields , 173(3-4):859-929, 2019. ISSN 0178-8051. doi: 10.1007/ s00440-018-0845-x. URL https://doi.org/10.1007/s00440-018-0845-x .

Cosme Louart and Romain Couillet. Concentration of measure and large random matrices with an application to sample covariance matrices. arXiv preprint arXiv:1805.08295 , 2018.

Alaa Saade, Florent Krzakala, and Lenka Zdeborová. Spectral clustering of graphs with the bethe hessian. Advances in Neural Information Processing Systems , 27, 2014.

Benjamin Aubin, Bruno Loureiro, Antoine Maillard, Florent Krzakala, and Lenka Zdeborová. The spiked matrix model with generative priors. Advances in Neural Information Processing Systems , 32, 2019.

Marco Mondelli and Andrea Montanari. Fundamental limits of weak recovery with applications to phase retrieval. In Conference On Learning Theory , pages 1445-1450. PMLR, 2018.

Marco Mondelli, Christos Thrampoulidis, and Ramji Venkataramanan. Optimal combination of linear and spectral estimators for generalized linear models. Foundations of Computational Mathematics , 22(5):1513-1566, 2022.

Ramji Venkataramanan, Kevin Kögler, and Marco Mondelli. Estimation in rotationally invariant generalized linear models via approximate message passing. In International Conference on Machine Learning , pages 22120-22144. PMLR, 2022.

Marco Mondelli and Ramji Venkataramanan. Approximate message passing with spectral initialization for generalized linear models. In International Conference on Artificial Intelligence and Statistics , pages 397-405. PMLR, 2021.

Greg W. Anderson, Alice Guionnet, and Ofer Zeitouni. An introduction to random matrices , volume 118 of Cambridge Studies in Advanced Mathematics . Cambridge University Press, Cambridge, 2010. ISBN 978-0-521-19452-5.

A Proof of Theorem 1.2

We now provide a rigorous proof of the result that was sketched in Section 2. This proof is essentially identical to [Deshpande et al., 2015, Lemma 4.4]. Recall the iterates (30) given by s 0 = x 0 -µ 0 ⊙ x ⋆ and

where µ t = ( µ t i ) i ≤ N is given by the recursion

and x ⋆ = ( x ⋆ i ) i ∈ [ N ] is a vector with independent coordinates distributed according to P 0 . By Lemma 2.4, for each a ∈ [ q ] , and any pseudo-Lipschitz function ϕ : R → R ↦→ R we have that almost surely

as was defined in (18). For any pseudo-Lipschitz function ψ : R → R , we have ϕ ( x, y ) = ϕ ( x -µ t a y ) is also pseudo-Lipschitz, so (44) implies that

almost surely.

Now let x t be the iterates from the spiked AMP iteration for the inhomogeneous Wigner matrix (24) we derived in (26)

It now suffices to show that for fixed t and all a ∈ [ q ] that

almost surely. This will imply that s t i + µ t a x ⋆ i and x t i have the same asymptotic distribution which finish the proof of Theorem 1.2 by (45).

We now prove (47). Since ψ is L -pseudo-Lipschitz we have

The Cauchy-Schwarz inequality implies that

where s t a = ( s t i ) i ∈ C a N ∈ R | C N a | , x t a = ( x t i ) i ∈ C a N ∈ R | C N a | . Therefore, to prove (47) it suffices to prove that for all t ≥ 0 ,

where

Clearly, if we initialize x 0 , s 0 at zero then (48) and (49) are satisfied by our state evolution equations (5). Notice that (49) follows directly from (45) applied to the square function. We use here that we assumed that the second moment of x ⋆ is finite.

We now focus on proving (48) through strong induction. By definition of the iterates (43) and (46),

where [ · ] i corresponds to the i th row of a vector and b x t -1 and b s t are the Onsager terms defined in (14) with respect to x t -1 and s t -1 respectively. The Cauchy-Schwarz inequality and Jensen's inequality imply that there exists some universal constant C such that

We now control each term separately.

To control the first term, notice that the matrix 1 N [ 1 √ ∆ ⊙ A ] has iid entries within blocks and the sizes of the blocks diverge with the dimension, so we can control the sums of the squares of within each block using standard operator norm bounds Anderson et al. [2010]. The first term vanishes in the limit because f is pseudo-Lipschitz so we can apply the induction hypothesis bound which controls (48) at time t -1 .

To control the second term, notice that for i ∈ C N a by Lemma 2.4 applied to the pseudoLipschitz function yf t -1 ( x ) that

almost surely. This implies that the average of such terms vanishes since we assumed that the second moment E [ x ⋆ 0 ] 2 is finite.

To control the third and fourth terms, we can expand the definition of the Onsager terms and use the assumption that f ′ is pseudo-Lipschitz and almost surely bounded. Both terms vanish because our strong induction hypothesis gives us control of (48) at time t -2 .

Since all terms vanish in the limit, we have proven (48) for all a ∈ [ q ] , which finishes the proof of statement (47) and the proof of Theorem 1.2.

B Comparison with a naive PCA spectral method

In this appendix, we wish to show how the spectral method we propose differs, in practice, from a naive PCA. We provide an example of the spectrums of Y and ˜ Y before and after the transition at

35

35

30

30

25

25

20

20

15

15

10

10

0

0

150

100

-50

0

50

100

150

-100-75-50-25

。

25

50

75

100

35

35

30

30

25

25

20

20

15

15

10

10

5

5

0

0

2.52.0-1.51.00.5

0.0

0.5

1.0

5

4

-3

2

1

0

1

SNR(∆) = 1 . In Figure 3 there is no clear separation of the extremal eigenvalue of Y from the bulk around this transition. This is in contrast to Figure 4 where there is an extremal eigenvalue of ˜ Y appearing at value one.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers approximate message passing algorithms for reconstructing a rank-1 signal when corrupted by a symmetric matrix of noise with a block-variance structure; it is assumed the signal _x^*_ has iid coordinates generated from a prior distribution.  One then forms the matrix $Y = x^* (x^*)^T/\\sqrt{N} + A \\odot \\sqrt{\\Delta}$, where $\\Delta$ is a blockwise constant, positive matrix.  The matrix $A$ is a symmetric gaussian matrix with iid off-diagonal entries of size $N \\times N$.   \n\nThe number of blocks $q$ appears to bounded independent of matrix size, and it is assumed one has access to the scale matrix Delta.\n\nThe first algorithm is a proper generalization of an AMP recursion for the case of a rank-1 signal with Wigner noise.  The state evolution is shown to converge (Theorem 1.2 -- using a modification of existing techniques) and correspond to the solution of the Bayes optimal estimator (Theorem 1.4).  \n\nThe spectral method designs a linear recurrence which is (locally? and conditionally?) optimal, in that it also recovers the same estimator by computing a principal eigenvector of an associated matrix.\n\nSTRENGTHS:\n1) The mathematical content is sound.  The main content (theorem 1.2) is proven in the supplemental material.  This adds an algorithmic aspect to a model which has attracted recent information theoretic attention.\n2) The paper presents an algorithm which is optimal for the problem posed.  The algorithm is part of a larger well-studied class.\n3) The performance of the algorithm is illustrated numerically in a simple case.\n4) A linear method is presented, which may reproduce the more complicated general AMP performance for the hidden spike.\n\nWEAKNESSES:\n0) The article does not provide any broader context for the technical results it develops; all but the first few pages are technicalities related to AMP theory.  It is targeted at experts in approximate message passing, and it does not develop much of the information-theoretic aspects of the model (which I gather are proven in Guionnet et al).  There is no conclusion.  There is nothing in the way of practical considerations or relations to application (although I would say this alone is forgivable, if the paper were otherwise immaculate).  Much of the main text is occupied by technicalities related to the formulation of the main theorem and in summoning relevant AMP theory from Javanmard and Montanari as well as Deshpande et al.  \n1) The main theorem (1.2) is an adaptation of an existing result.  Moreover, approximate message passing algorithms are well studied and many theorems exist for them.  The presence of non-iid noise makes it somewhat unique, but I think it is fair to say that this is not a big extension of existing theorems.  (In particular, there is a change of variables to connect the homogeneous and inhomogeneous cases).\n2) The spectral method, which is introduced, is largely left half-baked.  There is a conjecture (1.6) related to it, and there is a 1 page description of how the method is developed.  There is an equation (42) showing that the overlap evolution is unstable when a certain Perron-Frobenius eigenvalue is larger than 1.  But finally, the main points here are left as conjectures.\n\nQUESTIONS:\nMajor:\n\n0) Much of the valuable main paper space is used on technicalities of how one AMP is formally recast as another.  What conclusions should be drawn from this?  What is the context of the information-theoretic properties of the model?  What is the surprise of the linear method (mentioned briefly in the third bullet at the start?).  Lots of needed context is missing from this paper.\n1) The spectral method proposed at the end would seem to suggest that there is, finally, no need for the AMP algorithm (spectral methods are wildly simpler).  However the actual claims about what the spectral method does are left quite implicit (reading section 3).  In simple language, what do you conjecture happens for the spectral method above/below what transition?  A simple theorem in section 3 might be appropriate.   \n2) What is the recovery transition, and why should the spectral method be optimal?  (or what more than optimism justifies Conjecture 1.6)?  Good numerical results would be appropriate.\n3) The version of the AMP algorithm that solves the inhomogeneous spiked Wigner problem is finally presented in (32).  The assumptions that you formulate are not clearly directly relevant to this theorem (i.e. you could formulate your algorithm without all the heavy assumptions).  This means that in some sense these assumptions are really technicalities suited for those who want AMP details and/or proof details.  What are the minimal assumptions you need for actually solving the inhomogeneous spiked Wigner model? \n4) A discussion of the extent to which it is possible to actually implement the AMP iteration for the posterior would add to the broader appeal.\n\nMinor:\n* What is t in Assumption 1.1 part 2.  Does the family include all t?  Also this sentence is malformed.  What is the assumption? \n* What is M in (4)?\n* Eq (11) contains an $x_j$ as yet unused?\n\nLIMITATIONS:\nThe assumptions are clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the (symmetric) rank-1 matrix estimation problem with inhomogeneous noise. Here inhomogeneous noise refers to a symmetric noise matrix that is block-wise constant where the number of blocks is a constant relative to the dimension. \nThis paper proposes an approximate message passing (AMP) algorithm and shows the corresponding state evolution result. \nAnother piece of contribution is the design of a spectral algorithm that outputs the principal eigenvector of a rescaled and recentered matrix. \nNumerics suggest that this outperforms the naive estimator of the principal eigenvector of the data matrix per se.\n\nSTRENGTHS:\nThe most interesting (at least to me) part of the paper is Section 3 where a nonstandard spectral algorithm is introduced and analyzed to some extent. \nAs the authors commented, this estimator outperforms (at least numerically) the naive one corresponding to the original matrix Y. \n\nAnother satisfactory aspect of the result is the coincidence between the fixed point of AMP and that of the Bayes-optimal estimator (i.e., E[x^* | Y]), though this is not surprising.\n\nWEAKNESSES:\n1. The majority of the paper is devoted to AMP and its state evolution whose proof is a rather standard reduction to the matrix-valued AMP by Javanmard--Montanari. I didn't check the details carefully since everything goes as expected. But it's still good to see things written down formally. \n\n2. Section 3 is interesting at a heuristic level. However, I have a doubt regarding the authors' claim. \n\nIn line 131, it is claimed that \"we rigorously show that with SNR<1 our proposed spectral method fails to recover the signal\". I don't think the analysis in Section 3 constitutes a proof of this claim. \nIt was shown that the trivial fixed point of an AMP with linear denoiser is attractive when SNR<1. I agree with this, but this does not imply that the asymptotic overlap of the spectral estimator is 0 when SNR<1. \nThe iterate of the linearized AMP converges (in constant number of steps) to the principal eigenvector (i.e., the spectral estimator) only when a spectral gap is present. \nWhen SNR<1, there is no spectral gap and it is unclear how the iterate of linearized AMP is related to the principal eigenvector. \nIt may converge to some other vector, depending on the initialization. \nIn fact, rigorously speaking, I think it's fundamentally unlikely to prove subcritical behaviour by exploiting linearized AMP. \nThe analysis only proves the attraction of 0 when SNR<1, which is an *evidence* that the phase transition threshold is 1.\nHowever, this implies neither \"spectral fails when SNR<1\" nor \"spectral works when SNR>1\". \n\n3. If I understand correctly, the Delta matrix is assumed to be *known*. A very important aspect that was not discussed at all (correct me if I'm wrong) is what happens when Delta is unknown which appears (to me) to be a slightly more realistic assumption. \nIn that case the Bayes-AMP is no longer a practical algorithm (even with warm start) and the spectral algorithm is also not computable. \nIn fact, is it fair to say that the proposed spectral algorithm outperforms the naive one *because* it uses Delta information?\nCan the price of lacking the knowledge of Delta be quantified?\nI know this may go beyond the scope of the present paper. \nBut it seems to be an interesting nontrivial point that's worth mentioning/discussing.\n\nQUESTIONS:\n1. Line 7, \"info-theoretic optimal Bayes fixed-point equations\" sounds like a confusing phrase. Consider expanding it as \"the FP equations satisfied by the Bayes-optimal estimator\". \n\n1. Line 17, replace \"etc...\" with \"etc.\"\n\n2. Line 20 \"block-constant Gaussian noise\" is mildly confusing (sorry for being pedantic). The noise is never constant; consider saying \"Gaussian noise with block-constant variance profile\" or something like that. \n\n3. Line 46, is the word \"detect\" properly used? Or do the authors mean \"weak recovery\"? My understanding is that they are not the same question. \n\n3. I couldn't parse the grammar of 2. of Assumption 1.1.\n\n5. L in the definition of PL functions seems to be a floating parameter. In standard theory, L is tied to the regularity of prior distributions and initializers. This needs to be quantified. \n\n5. There are multiple mistakes in Remark 1.3. The correct statement should be \"convergence under all L-PL functions is equivalent to convergence in Wasserstein-L\". Also, bounds on k-th (k>2) moment is a *stronger* assumption than bounds on 2nd moment, consequently, convergence in W_k is *stronger* than W_2. \n\n5. In Equation (6), has the notation E_{posterior} been defined? If not, then this notation is definitely too sloppy. \n\n5. Line 137, there are redundant spaces in \"N - vector\". \n\n5. Line 141, please explicitly write i in [N] when introducing the notation f_t^i. \n\n5. Equation (33), I'm not familiar with the physics jargon \"Nishimori identity\" and didn't check Lelarge--Miolane, but isn't this just the law of total expectation (https://en.wikipedia.org/wiki/Law_of_total_expectation)?\n\n5. In Section 3, an AMP with denoiser f_t(x) = x is considered. Why is identity denoiser a good thing to consider, besides its simplicity? I understand this comes from linearization. But for the sake of being self-contained, I suggest comment on this. \n\n4. This paper handles heterogeneous noise. Is it true that with essentially no additional effort, heterogeneous prior can also be handled?\nAt least when the partition for the prior is the same as that for the noise? \nI didn't check the details and may be wrong. \n\n5. The number of blocks q is assumed to be a constant relative to N. What happens if it grows with N? Clearly if q=N nothing can be said. But this question makes sense for sufficiently slowly growing q. After the reduction to matrix-valued AMP, the SE result of Javanmard--Montanari is no longer applicable since they require the width of the matrix iterates to be a constant. Could the authors comment on what can potentially happen here?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an analysis of an AMP algorithm for the spiked Wigner model with inhomogeneous noise. The paper builds on the matrix AMP framework to derive the state evolution equations for the considered AMP recursion for the studied model. The paper further shows that if the denoising functions are the Bayes one, then the fixed point equation of the state evolution of the AMP algorithm is the same as the one satisfied by the Bayes optimal estimator.\n\nThe paper also leverages the developed machinery to study the properties of a spectral algorithm which is motivated by considering the the identity denoising functions. It is conjectured that this spectral algorithm exhibits optimal phase transition.\n\nSTRENGTHS:\nI find the paper to be generally well-written and it is not hard to follow. The problem that is considered is interesting and the presented results generalize previously known results for the spiked Wigner model with homogenous noise to the non-homogenous case.\n\nWEAKNESSES:\nMinor comments:\n- Page 1: Please state whether $\\tilde{\\Delta}$ and/or $g$ are assumed to be known. From Eq.(3) it seems that we do assume that $\\Delta$ is known.\n- Page 2, line 82: The notation for $f_t:\\mathbb{R}^N\\times \\mathbb{N} \\to\\mathbb{R}^N$ is confusing/informal because from the displayed equation, it seems that $f_t$ takes input from $\\mathbb{R}^N$. Is $t$ supposed to be the input integer in $\\mathbb{N}$?\n- Page 2, line 83: If $f_t^a$ are general Lipschitz functions, it is not clear to me why $f_t$ is linear.\n- Page 3, line 112: It seems to me that if we replace the second moment assumption by finite $k$-th moment, we are strengthening the assumption, not weakening it.\n\nQUESTIONS:\n- In the abstract it is claimed that the spectral method is shown to match the information-theoretic transition. If I understand the arguments of the paper correctly, it seems that this is only a conjecture that is based on a heuristic, and no rigorous proof was given. Did I miss anything?\n\nLIMITATIONS:\nNo concerns regarding potential societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the spiked Wigner problem with inhomogeneous noise, i.e. the inverse problem of estimating a rank-matrix through an inhomogeneous noise channel. This problem naturally arises in many applications and a universality result makes the problem considered quite general with regards to the noise distribution.\n\nThe authors have made several contributions in this paper:\n1. They have derived the AMP recursions to solve the spiked Wigner problem.\n2. The most interesting property of AMP-like methods is that their behavior can be characterized exactly through a set of low-dimensional state evolution equations. This paper obtains the state evolution for the AMP recursion that solves the problem considered.\n3. The authors analyze the AMP algorithm with identity denoisers and show that it corresponds to an spectral method for a specific matrix. More interestingly, the authors conjecture that this linear version of AMP detects a spike in the same region as the general AMP.\n\nSTRENGTHS:\n- The paper considers an interesting problem with many applications. \n- The authors derive an AMP method to solve this problem.\n- The authors obtain the state evolution of this AMP recursion, thus fully characterizing the macroscopic behavior of the AMP recursion at **each** iteration of the algorithm and **not** just the fixed points. This result gives us a **theoretical** way of obtaining the estimation error of the AMP method using very general metrics\n- The authors show that when the Bayes optimal denoisers are used (i.e. the mean of the posterior) the fixed point equations exactly match the Bayes optimal fixed point equation of another recent work.\n- Finally, the authors analyze the linear version of the AMP algorithm and show it is equivalent to spectral method for a specific matrix.\n\nTo summarize, the authors look at the spiked Wigner problem with inhomogeneous noise, derive an AMP method to solve it, and fully characterize the theoretical behavior of the AMP method in a certain high-dimensional asymptotics. This is in contrast to many other methods that are used in practice, but have no theoretical guarantees or guarantees of the form of high-probability upper bounds on the error in certain metrics. AMP methods, allow us to obtain the **exact** error in many **different error metrics** in a certain **high-dimensional limit**.\n\nWEAKNESSES:\nThe weaknesses that come to my mind are mostly the usual weaknesses of the AMP algorithms:\n- The AMP algorithms are often described as not very useful to solve problems in practice due to their instability often requiring a lot of tweaks such as damping to make them converge. I do not see much comments in this work regarding the stability of the AMP method described. This is in part due to generality of the algorithm (described using general denoisers) and in fact for the linear case the authors show a condition for convergence, however no mention of convergence or potential issues in the general case is mentioned. I should admit however that for example Bayes optimality of the AMP method with Bayes optimal denoisers makes this method interesting in practice for this problem.\n- That being said, AMP having theoretical guarantees would still be very valuable as a theoretical tool. However, obtaining the errors through the state evolution equations are often very nontrivial due to the need to calculate expectations in a recursive formula that often need MCMC methods and are sometimes computationally not much better than running AMP for several instances of the problem and estimating the final error.\n- As mentioned above, the AMP results only hold in a certain asymptotic regime. However in practice, very good match is observed even for moderately sized problems as the authors also mention.\n\n## Minor comments\nI believe the work is hard to follow for someone who is even familiar with the problem considered but not familiar with the AMP literature. It assumes the readers are knowledgeable in this area and very familiar with previous works which makes it hard to follow.\n\nQUESTIONS:\n- It would be nice if more explanations are added regarding the convergence of the AMP algorithm, any issues it might have in practice and the usefulness of the state evolution equations as they are recursive equations that do not seem trivial to simulate. For example, the convergence of AMP, or uniqueness of its fixed points (if it is in fact unique) should either be shown or it should be clearly mentioned that these are open questions that need to be addressed in order for this AMP method to be fully characterized.\n- In many AMP papers, the authors make a correspondence between the AMP algorithms and estimators that are more familiar to the community. For example, using certain (generalized) proximal operators as the denoisers, the AMP would then can be shown to have fixed points that are also critical points of a certain loss function, i.e. AMP would be doing a form of M-estimation. Similarly, one could make a correspondence between other Bayesian estimators such as MMSE estimator and denoisers that use the posterior distribution such as the Bayes optimal one also mentioned in the paper. Can such results be shown for the AMP algorithm considered here? For example showing that AMP with a certain denoiser is minimizing a certain loss function, etc.? Since people are much more familiar these optimization based estimators, it would make this paper a lot easier to understand and follow. Similarly for the Bayesian estimators which are trying to estimate the signal based on the posterior, such as MMSE estimator, it would be good to make the correspondence between AMP and such estimators more clear.\n\nLIMITATIONS:\n- The practical limitations of the AMP algorithms and how easy/hard it is to use the state evolution in practice are not adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}