Title: The Franz-Parisi Criterion and Computational Trade-offs in High Dimensional Statistics
Authors: Afonso S Bandeira
Ahmed El Alaoui
Samuel B. Hopkins
Tselil Schramm
Alexander S Wein
Ilias Zadik
Venue: NeurIPS 2022
Keywords: Information-computation gaps
low-degree likelihood ratio
statistical physics
MCMC methods
sparse regression

Abstract: Many high-dimensional statistical inference problems are believed to possess inherent computational hardness. Various frameworks have been proposed to give rigorous evidence for such hardness, including lower bounds against restricted models of computation (such as low-degree functions), as well as methods rooted in statistical physics that are based on free energy landscapes. This paper aims to make a rigorous connection between the seemingly different low-degree and free-energy based approaches. We define a free-energy based criterion for hardness and formally connect it to the well-established notion of low-degree hardness for a broad class of statistical problems, namely all Gaussian additive models and certain models with a sparse planted signal. By leveraging these rigorous connections we are able to: establish that for Gaussian additive models the "algebraic" notion of low-degree hardness implies failure of "geometric" local MCMC algorithms, and provide new low-degree lower bounds for sparse linear regression which seem difficult to prove directly. These results provide both conceptual insights into the connections between different notions of hardness, as well as concrete technical tools such as new methods for proving low-degree lower bounds.

TL;DR: We establish some rigorous connections between two different frameworks for computational hardness of statistical problems: algebraic methods based on low-degree polynomials and geometric methods rooted in statistical physics.

Review discussion:
### Originality
The papers contains several novel results: the definition of the FP criterion, the study of the relationship of FP hardness with LD hardness and MCMC hardness in different models and new bounds for sparse linear regression.
Moreover, the proof of the bounds relies on the FP criterion, which has to be considered not only as a conceptual item, but also as a practical tool for proofs.
The authors clearly and convincingly state the relationship of their results with existing literature in many remarks both in the main text and in the supplementary materials.
I am not familiar with the literature enough to assess whether all relevant work has been adequately cited.
### Quality
The authors clearly state the assumptions underlying their results, and provide seemingly correct proofs.
I did not check the details of the proofs.
### Clarity
The manuscript is overall clearly and nicely written, and together with the extensive supplementary material provides enough information for the expert reader to be able to fully assess correctness and to reproduce the proofs.
### Significance
I believe that the results presented in this manuscript related to FP-hardness are significant.
Linking algorithmic hardness with statistical-physics-like properties of computational problems is a long-standing research effort. As far as my knowledge goes, it is still not completely clear how geometric properties of free-energy landscapes, such as replica symmetry breaking, relate to computational hardness, and this seems a promising step.
I am not qualified enough to properly assess the significance of the results on sparse regression, and on the novel proof technique based on the FP-criterion.
I am curious on how the FP hardness criterion relates to overlap gap properties and their consequences: do the authors have some intuition that could be added to the manuscript?
In the manuscript the authors hint at some relationship between the hypothesis testing setting they study and the (to me) more common estimation (retrieval of the planted signal) setting: is there some more precise relationship between computational and theoretical thresholds of the two settings?
The authors clearly assess the limitations of their results in the supplementary materials, where they provide counterexamples that suggest that their results do not point towards a complete equivalence between low-degree hardness and FP hardness.
These limitations do not undermine the validity or significance of their results.
No
We thank the reviewer for the positive comments and encouraging feedback. We are also very excited to make a first step on the "long-standing research effort" of "linking algorithmic hardness with statistical-physics-like properties of computational problems".
"I am curious on how the FP hardness criterion relates to overlap gap properties and their consequences"
That is a very good question. In many inference settings, the so-called overlap gap property (OGP) for inference has been a useful tool in predicting/proving the failure of low-temperature MCMC/local-search methods to estimate a planted signal.
Indeed, OGP has played a key role in the low-temperature MCMC lower bounds for sparse regression [GZ22] and sparse PCA [BWZ20].
Comparing now the FP and OGP, notice that the two predictions differ fundamentally in that they refer to two *different tasks*: FP predicts the hardness of detection/hypothesis testing, while OGP is tailored to the hardness of estimation tasks (see also below on that). No version of OGP has yet been considered in the context of detection.
Despite that, one can naturally wonder whether, for the multitude of models where detection and estimation have similar behavior, one can relate the two predictions. This would have been very interesting, but at the moment we do not have a rigorous connection between OGP and FP hardness even under such an assumption.
Yet, our intuition is that the two predicted hardness thresholds can significantly differ. To make a case in point, let us consider tensor PCA with a uniform signal on the hypercube [RM14], which is a Gaussian additive model where detection and estimation is believed to have the same behavior. In this model, it is well-known in the low-degree hardness literature that the currently predicted "computational limit" to estimate the signal coincides with the LD-hardness prediction.
Hence, from our "FP-LD equivalence" Theorems 4.3-4.4 it also coincides with the FP hardness prediction.
Yet, it is known that in tensor PCA zero-temperature local search/gradient descent methods fail at a much different "local" threshold [BGJ20].
While OGP has not yet been analyzed for tensor PCA, judging on the prior work, it is natural to conjecture that it appears exactly at this "local" threshold and not at the "computational threshold". Such an expected result will yield indeed a much different hardness OGP prediction compared with the FP one. It is natural to predict that this phenomenon should be generically true whenever MCMC methods underperform in estimating the planted signal as compared to other methods (see also [CMZ22]). Making the above observations rigorous is a very interesting direction for future work. We plan on adding such a discussion in the next version of our paper.
"...the hypothesis testing setting they study and the (to me) more common estimation (retrieval of the planted signal) setting: is there some more precise relationship between computational and theoretical thresholds of the two settings?"
This is also a very good question raised by the reviewer.
While in many natural settings the detection/estimation thresholds coincide (or can at least be related), it is unfortunately not true that the two are equivalent.
For example, when the prior is sufficiently asymmetric, detection is often strictly easier than estimation. Consider for example, the case of non-negative sparse PCA: the noise is $Z \sim N(0,I_n)$, the signal $x$ is $k$-sparse with coordinates $0$ or $1/\sqrt{k}$ and we observe $Y=\lambda x^{\otimes 2}+Z.$ When $k \gg \sqrt{n}$, polynomial-time/information theoretic (IT) detection is possible for a much wider range of $\lambda$ than polynomial-time/IT estimation.
One can then naturally wonder whether (poly-time/IT) estimation implies (poly-time/IT) detection. Unfortunately, there are pathological examples that make such a claim false in full generality (see Section 3.4 of [BMV+17]).
However, in most natural settings this direction holds: using the specifics of the studied model it is usually possible to establish a simple reduction proving that any (poly-time) algorithm that solves estimation can be made into a (poly-time) algorithm that solves detection (see e.g. [Appendix C, CGHK+22]). Characterizing a concise, general set of assumptions under which such a reduction holds is an interesting and fundamental question of future work.
**All references are from the citation list in the supplementary material.
I thank the authors for taking the time to answer my questions.
I agree with the authors that adding a brief discussion about the link between FP/OGP and detection/estimation in their paper (or in the SM) would be beneficial to the reader.
I have no further questions for the authors, and I confirm my previous evaluation.
## Summary of evaluation:
I was not able to verify the math in this timeframe.
However, the results seem plausible, since the tradeoffs obtained here for various problems are expected to be true. The contributions in this work are conceptually interesting and raise potential directions for further research.
However, the actual results obtained seem a bit weak since they require a lot of assumptions.
Also, they are not readily applicable to other problems due to the difficulty of the calculations involved.
Moreover, the proofs are highly technical and will take quite a bit of time to absorb for even experts in either statistical physics or statistics, let alone the average NeurIPS reader, which leads to concerns regarding the fit.
## Strengths:
1. Lower bounds against computational methods have long been studied from different perspectives, especially for average-case problems where NP-hardness results are not known. Recently, researchers have been trying to build bridges between different perspectives, e.g. Brennan et al. 2021 show that SQ algorithms and low-degree tests are equal power under some assumptions.
Likewise, this work attempts to bridge the gap between statistical physics approaches and low degree tests.
2.
Although the LD test became useful for its ease-of-use in predicting computational thresholds, the authors state that it's hard to apply it to certain parameter regimes of sparse regression. They overcome this with their new FP criterion.
3. The FP criterion recovers predicted thresholds for the spiked Wigner model of PCA. The example worked out is very helpful to understand it.
4. Formalizing all of the intuition this work brings about is nontrivial. The technical mastery in this work is commendable.
## Weaknesses:
1. The main weakness is that even if the FP criterion is a potentially useful criterion, it still seems quite hard to use it for other problems of interest.
For example, even for the application to planted sparse models, the authors have to spend significant effort to bound the FP criterion, such as conditioning on a high probability event (definition 6.3).
So, the usefulness of this criterion itself for other problems seems rather limited and technically tedious even for experts.
2. Based on my understanding, the transitive symmetry property seems to break when we consider sparse PCA where the sparsity is *at most* k, instead of *exactly* k.
Therefore, the results here do not apply for this simple variant of sparse PCA since transitive symmetry seems to be a rather strong assumption. Could the authors clarify this?
3.
Although the results are intriguing, my main concern is regarding the fit of this paper for this conference. Due to its length and non-triviality, It may be better suited for a journal or a more theory-focused conference.
4. For someone more from an ML background and not from a statistical physics background, it can be hard/impossible to follow section 1.3 in the appendix.
Despite this, I understand that it is not easy to have a self-contained intro to the statistical physics terms used here. I'd suggest that a good reference should be cited to learn these better for the interested reader.
5.
In the end of page 14 of the appendix, it's stated that very few tools are available to prove the success of MCMC methods for inference. If that's the case, then why are hardness against MCMC methods a good proxy for computational hardness?
A few questions were raised above.
## Other questions/comments:
1. Just to clarify, can the sample size in theorem 6.2, i.e. m = (1 + o(1))Rk log(n/k), be replaced by m <= (1 + o(1))Rk log(n/k)? It seems a bit unusual to set the sample size to be something specific
2. L86: "precisely" -> "precise"
3. L166: the partition function is not defined, I believe the first place it's defined is L271.
4. In the last paragraph of page 7 of the appendix, "says bounded" -> "stays bounded"
This is primarily a theoretical paper. Limitations of the results are sufficiently addressed, including the various assumptions and conceptual limitations (section 4 of the appendix). Negative societal impacts are not discussed, but I do not think there is any.
No
We thank the reviewer for their thought-provoking comments.
We address the issues raised one by one.
"This work deals with proving algorithmic hardness of a class of statistical problems against local MCMC algorithms"
From our perspective, the main point of our work is connecting low-degree hardness and notions of hardness arising from free energy landscapes in statistical physics.
Though properties of the free energy landscape can be used to rule out MCMC/gradient-based algorithms, the FP criterion is not equivalent to local MCMC algorithms, and, we believe, carries richer information about problem structure.
"The main weakness is that even if the FP criterion is a potentially useful criterion, it still seems quite hard to use it for other problems of interest."
We believe the FP criterion is actually a *simpler* way to prove low-degree hardness than the alternative direct approach, and we see this as a major strength rather than a weakness. FP can be applied to a wide range of problems; some will be easy to analyze (e.g. spiked Wigner) and others harder. Sparse regression is a difficult problem, and while the FP approach is complicated, it is still much simpler than the alternative approach would be. For some problems (like sparse regression and subsequent work on group testing [CGHK+22]), FP is the *only* way we know how to prove optimal low-degree lower bounds, and we feel this speaks to the value of the FP approach.
"Therefore, the results here do not apply for this simple variant of sparse PCA since transitive symmetry seems to be a rather strong assumption"
The reviewer is correct in pointing out that the assumptions of our meta-theorem for "low-degree hardness => MCMC lower bounds" rules out the "at most k" sparsity constraint on the prior.
On the other hand, this is typical of meta-theorems; at the expense of making assumptions, we are obtaining a general connection between low-degree bounds and MCMC methods (MCMC is a simple and natural class of algorithms not obviously captured by the low-degree framework). What excites us in our meta-theorem is the fact that it captures, with a single argument, many interesting cases that have been studied individually (e.g. [BWZ20] for sparse PCA). We hope that this will be a stepping-stone towards a better understanding, in a unified fashion, the phenomena of statistical-to-computational gaps, and obtaining less restrictive assumptions is an important direction for future research.
"Although the results are intriguing, my main concern is regarding the fit of this paper for this conference. Due to its length and non-triviality, It may be better suited for a journal or a more theory-focused conference."
We feel that the core ideas in the paper, such as the definition of the FP criterion, are relatively simple to explain and would be appreciated by the NeurIPS community, where connections between ML/inference and statistical physics is a vibrant topic of interest (see e.g. Maillard-Loureiro-Krzakala-Zdeborova, NeurIPS'20 and many others) and results on MCMC methods in inference are traditionally discussed. We agree that our sparse regression example is fairly technical; we intentionally chose a well-known hard problem in the low-degree literature which has resisted analysis so far (and where non-trivial arguments are probably unavoidable) to illustrate the full power of the FP criterion. We feel that a reader need not follow the full details of this calculation in order to appreciate the main ideas presented in the paper.
"I'd suggest that a good reference should be cited to learn these better for the interested reader."
Thank you for pointing this out. We will add some references (e.g. the book of Mezard-Montanari "Information, Physics, and Computation").
"few tools are available... why are hardness against MCMC methods a good proxy for computational hardness?"
Indeed, lower bounds against MCMC methods should perhaps not be thought of as strong evidence for inherent computational hardness.
Instead, MCMC is a simple and natural class of algorithms that are widely applicable, easy to implement and popular in practice, and so MCMC in inference has attracted much study (even in situations where the MCMC methods in question perform worse than the best-known poly-time algorithms). We also want to stress again that the FP criterion is not equivalent to hardness for MCMC, and as we show in some cases it is likely stronger (since it is equivalent to LDLR for many models).
"Just to clarify, can the sample size in theorem 6.2, i.e. m = (1 + o(1))Rk log(n/k), be replaced by m <= (1 + o(1))Rklog(n/k)?"
Adding more samples can only make the problem easier. The theorem includes both an upper bound and a lower bound. For the upper bound (algorithmic result), it would be fine to replace m = ... by m >= ..., and for the lower bound it would be fine to replace m = ... by m <= ....
**All references are from the citation list in the supplement.
I thank the authors for their response. Since they have addressed some of my concerns, I'm willing to raise my score. I believe that as a reader of this work, it's clear that FP is not equivalent to MCMC (please clarify if something in my review seemed to suggest otherwise, happy to edit for the sake of correct exposition).
Also, as a reader, I felt that the main contribution was the computational hardness conclusion, thanks for clarifying that it is in fact the connection between LD hardness and statistical physics hardness.
Thank you very much for raising your score. (No need to edit the review -- we just wanted to emphasize that the main point is not just MCMC lower bounds.)
I find the paper to be very interesting, and am strongly supportive of its publication in NeurIPS. Some of the insights are:
(1) An annealed approximation to the Franz-Parisi free energy admits a second-moment type interpretation, and connects to the classical second-moment method in testing.
(2) A formal connection can be shown between restricting the geometric overlap and restricting the polynomial degree in the second-moment method. This seems to be particularly surprising in the planted sparse model setting of Section 3 where there is no explicit form for the low-degree likelihood ratio in terms of the overlap, and where the implication is shown from a symmetry condition alone.
(3) The connection can be used in both directions, to provide geometric approaches to analyzing the (conditional) low-degree likelihood ratio statistic, and to provide algebraic methods of analyzing the overlap landscape. It's helpful to have these connections elucidated at the level of generality of this current work, even if versions of these implications were used implicitly before in more specific models.
I guess the main weakness seems to be that it's unclear whether this FP-criterion fully captures the "physics intuition" for hardness in these types of problems, as (a) it is defined in terms of an annealed potential, and it's a bit unclear (at least to me) when this should have the same behavior as the replica-symmetric potential or the true quenched potential, and (b) it pertains to the properties of this potential only locally near 0, rather than to the entire overlap landscape.
But I think these issues are adequately discussed in Section 4, and in my opinion they provide food for thought and do not detract much from the strengths of the work.
I might suggest to the author(s) to include some more of the discussion in Section 1.3 of the supplement (on the Franz-Parisi potential) into the main text, if this is possible. Currently I find Remark 3.3 a bit hard to understand (i.e. what is the relation between the FP potential and a restriction of the partition function?). Without this, it's hard to grasp the intuition behind the main Definition 3.2, as well as how the local/global properties of Figure 1 should be interpreted in the discussions on page 5.
A few typos:
p2 l86, "its precisely domain"
p3 l121, "computational" is repeated
p5 l189, "local minimum separated from", minimum -> maximum
Yes
No
We greatly thank the reviewer for their careful reading and positive and encouraging feedback. We will make sure to add more details to Remark 3.3, borrowed from the discussion in Section 1.3 of the supplement, to make it more intuitive and accessible to the reader.

Meta-review:
role: AC
data: metareview: The paper establishes new bridges between two fundamental notions for the study of average case hardness of hypothesis testing problems: low-degree functions and the Franz-Parisi free energy, and also connect them to MCMC hardness. The paper has been very well received by the reviewers, who have done a great job. I agree with the reviewers on the quality of the paper. I also think that despite NeurIPS not being the most obvious avenue to submit such a paper, there indeed is an active sub-community of NeurIPS readers very much enthusiastic about such results, connections with statistical physics, etc. All issues raised seems to have been properly answered by the authors, so this is a clear accept for a high-quality paper full of rich notions and new fundamental connections.
role: PC

Decision: Accept