Title: Generative diffusion for perceptron problems: statistical physics analysis and efficient algorithms
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative diffusion

approximate message passing

perceptron

spin glasses

statistical physics

markov chain monte carlo
Abstract: We consider random instances of non-convex perceptron problems in the high-dimensional limit of a large number of examples $M$ and weights $N$, with finite load $\alpha = M/N$.
We develop a formalism based on replica theory to predict the fundamental limits of efficiently sampling the solution space using generative diffusion algorithms, conjectured to be saturated when the score function is provided by Approximate Message Passing.
For the spherical perceptron with negative margin $\kappa$, we find that the uniform distribution over solutions can be efficiently sampled in most of the Replica Symmetric region of the $\alpha$–$\kappa$ plane.
In contrast, for binary weights, sampling from the uniform distribution remains intractable. A theoretical analysis of this obstruction leads us to identify a potential $U(s) = -\log(s)$, under which the corresponding tilted distribution becomes efficiently samplable via diffusion.
Moreover, we show numerically that an annealing procedure over the shape of this potential yields a fast and robust Markov Chain Monte Carlo algorithm for sampling the solution space of the binary perceptron.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a theoretical extension of the analysis of the Stochastic Localisation framework through replica calculations. In particular, through the study of free entropy, one is able to asymptotically determine thresholds for sampling feasibility via AMP, being the latter conjectured to be optimal for the Bayesian denoising task in a high-dimensional setting. Furthermore, the authors apply their method to the perceptron model, in the spherical and binary weights setup. The spherical case is studied in the case of a uniform distribution on the solution space (non-convex), discussing AMP limits. The binary case is further explored: the authors show that with a peculiar choice of the potential $U(s)$ it is possible to circumvent the issue of a second maximum in the free entropy function, which would make AMP inapplicable. However, they stress that in the presence of structured data AMP would fail either, and it would be necessary to use MCMC; hence, they propose a potential $U_\\tau(s)$ that is a sort of regularised version of $-\\log s$ in order for MCMC to be applied and anneal $\\tau\\to 0$, showing such a procedure is more effective than standard $T$ annealing.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n- The content of the paper is original, and the claims made in the introduction are convincingly supported throughout the manuscript.\n- From a theoretical standpoint, the results are sound and well validated, even though they rely on the replica method, which the authors correctly acknowledge as “non-rigorous but extensively validated.”\n- The theoretical predictions are empirically verified in a synthetic setup that, while not of direct practical relevance to modern generative models, provides a reasonably novel benchmark for the application of replica methods in machine learning.\n- The authors are transparent about the limitations of the proposed theoretical and algorithmic methods, particularly regarding the use of AMP in realistic data scenarios, which is appropriately acknowledged in several parts of the paper.\n\n**Weaknesses**\n\n#### **1. Style and Structure**\n\nThe paper sits stylistically between a physics-oriented and a mathematically formal presentation. To improve clarity and accessibility:\n\n- If the authors wish to target the *physics-oriented style*, it would be helpful to enhance the intuition behind the derivations and the results, both theoretical and experimental. This should be feasible, as the treated examples are standard and well-known. This would also align better with a broad conference audience rather than a narrow community of replica theory experts.\n- If instead aiming for a more *mathematically rigorous style*, the paper should be revised to adopt a more formal structure, including clearly stated propositions, theorems, assumptions, etc.\n\nCurrently, the hybrid format makes the paper difficult to follow — even for readers familiar with replica theory but not embedded in the specialized community. In its present form, the structure is not well-suited to either tradition.\n\n#### **2. Theoretical Contribution**\n\n- The level of generality beyond *Ghio et al. (2024)* is not entirely clear. To highlight the originality of your contribution, it would be very helpful to explicitly state a proposition from *Ghio et al. (2024)* and then present your generalization as a formal result, ideally in a comparative form. This would greatly assist readers in understanding the novelty of your approach.\n- The discussion on MCMC is presented as a major contribution in both the abstract and the introduction, and it indeed forms an interesting bridge to the sampling community. However, the space dedicated to this result is extremely limited. I recommend expanding the explanation and clarifying its implications and connections to the rest of the work.\n\n#### **3. Presentation of AMP and Replica Results**\n\n- Section 4.2, which introduces AMP, feels isolated and incomplete. It could either be moved to the appendix or expanded to give a more self-contained and pedagogical explanation, especially clarifying the role of $\\phi_{{in}}$ and $\\phi_{{out}}$, which are introduced but never used later.\n- A similar issue arises with the replica computations (e.g., Equations 15–18): they occupy a significant amount of space, but are not discussed or interpreted in the main text. Either move them to the appendix or incorporate them into the narrative with interpretation and relevance.\n\nQUESTIONS:\n1. The paper’s style sits ambiguously between a physics-oriented and a mathematically rigorous presentation, making it difficult to follow for readers outside the core replica theory community. In this form the paper needs, in my opinion, a major reformatting, in accordance with the Limitations I described above. \n2. Key theoretical results lack formal statements (e.g., propositions or theorems) and are not clearly distinguished from prior work such as Ghio et al. (2024). Could the authors explicitly state these results in a formal way to better highlight the novelty and assumptions of their contributions? Even if they prefer a physics-oriented presentation, such highlighting would be beneficial.\n3. The connection to the broader machine learning community currently appears somewhat limited, being mostly confined to the introduction and a few remarks on AMP. In particular, the MCMC component—presented as a key contribution—is discussed rather briefly and remains somewhat disconnected from the wider context of modern sampling techniques in ML. It would significantly enhance the paper’s impact if the authors could elaborate on how their results relate to existing methods in the sampling literature, and clarify both the practical relevance and the limitations of their approach in more realistic ML settings. While additional examples are not necessary beyond those already included, a stronger effort to draw connections or analogies with practical ML applications would be greatly appreciated, especially given the acknowledged limitations of AMP and ASL in real-world scenarios.\n\nLIMITATIONS:\nYes, they recognise the limitations inherited from AMP and also from replica calculations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical framework to analyze the efficiency of sampling the solution space of high-dimensional random perceptron problems using generative diffusion algorithms. The authors employ a statistical physics approach, using the non-rigorous but well-validated replica method, to predict the performance of Algorithmic Stochastic Localization (ASL) , where Approximate Message Passing (AMP) serves as the score approximator.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n* The problem consists of novel formulation of perceptron problems. A key strength is the generalization of prior work (e.g., Ghio et al. [2024]) to handle unnormalized target densities.\n* The authors provide a detailed asymptotic analysis for both the spherical and binary perceptron models. The main contributions are explicitly stated and well-supported by the derivations.\n\n## Weaknesses\n\n* **Significance.** The method heavily relies on approximate message passing, which is a well-established concept but also has limited use in real-world application. The authors claimed that the method provides a fast enough algorithm for their \"synthetic\" setting (pp. 4). Therefore, I have a concern about its significance since I have the impression from literature that the solution acquisition method for perceptrons will not scale well. Since it relies on the quality of the sample of MCMC method, it is likely that the scalability of the proposed method is severely limited.\n* **Novelty.** Most of the concepts outlined in this paper are well-known and this paper does not produce new distinct theorems. Therefore, I believe the contributions of this paper are more in the empirical side, and I think a new formalism is in Eqs. (15-17) serve that purpose. However, the authors do not clearly articulate the reason why we look for Spherical Non-Convex Perceptrons. The novelty of this paper is not clearly expressed and the motivation of the work needs a comprehensive revision.\n* **Experiment.** The paper needs some experimental results on scalability, and the application side of spherical nonconvex perceptions. I also suggest the authors to include demonstrative illustrations and figures in Section 1 or 3, which help understanding of broad audience\n\nQUESTIONS:\n* Could you elaborate on the motivation for studying the spherical non-convex perceptron? While it is described as a model for \"glassy behavior\", what specific algorithmic or theoretical challenges does it present that makes it a crucial case study for your generative diffusion framework?\n* The τ-annealing MCMC scheme is proposed as a robust alternative to AMP for the binary perceptron. Could you provide a more detailed analysis of its scalability, perhaps by discussing its computational complexity and how its performance (e.g., time-to-solution) compares to other state-of-the-art heuristic solvers for this problem, especially as the system size N increases?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of sampling from a high-dimensional distribution \n$$p(\\mathbf{w}) = \\frac{\\psi(\\mathbf{w})}{Z},$$\nin a setting where only the unnormalized density $\\psi(\\mathbf{w})$ is known.\n\nThe authors study an approach based on generative diffusion (or stochastic localization), \nwhich allows to sample from $p$ by solving a stochastic differential equation (SDE). \nAt each time $t$, one has to compute the Bayes-optimal estimator of $\\mathbf{w}^\\star$ \nfrom the observation \nof $\\mathbf{y} = t \\mathbf{w}^\\star + \\sqrt{t} \\mathbf{g}$, where $\\mathbf{w}^\\star \\sim p$ and $\\mathbf{g} \\sim \\mathcal{N}(0,1)$. \nBased on existing literature, a natural approach to compute this BO estimator is via the Approximate Message-Passing (AMP) algorithm.\nUsing non-rigorous methods from statistical physics (namely the replica method), the authors show that whether this sampling procedure (called ASL -- Algorithmic Stochastic Localization) succeeds can be characterized \nfrom the study of a one-dimensional potential $\\phi_t(q)$, for which they provide an explicit form.\n\nThey apply this framework to perceptron-type problems:\nfocusing on the region where the problem is solvable, the goal is to sample uniformly from the set of solutions.\nFor the negative spherical perceptron, the analysis of the paper shows that ASL succeeds in (almost) all the replica-symmetric region of the phase diagram.\nIn the binary perceptron with zero margin, the authors show that instead ASL fails in the whole phase diagram, in agreement with previous works that argue computational hardness of sampling in the binary perceptron.\nOn the other hand, by tilting the measure over the space of solutions by a well-chosen potential, ASL is shown to succeed at sampling this modified measure in a large region of the phase diagram.\nFinally, sampling of this tilted measure is also shown numerically to be achievable by a MCMC algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nLet me clarify that I read through the main text in detail (following the computations and arguments),\nbut did not check the supplementary material for reasons of time.\nMoreover, while I am quite familiar with the statistical physics approaches to learning theory, this is slightly less the case for generative diffusion.\n\n**Strengths --**\n\n- The paper is written in a clear fashion. Concepts are exposed very clearly while \n  remaining concise (Section 2 is a good example), and I want to thank the authors for this effort in the writing.\n  The technical tools used here (the replica method and the analysis of AMP algorithms) are well presented and detailed. \n  The results are well described, e.g. via clear figures. \n- The replica computation in this generic setting, as well as the phenomenology of sampling in perceptron models \n  are contributions which would be of interest to the audience engaged in similar topics, as this a subject of strong recent activity and the paper \n  shows interesting results in this regard (e.g. that sampling a weighted measure over the space of solutions rather than the uniform one can make the problem drastically easier \n  in binary perceptrons).\n\n\n**Weaknesses --**\n\n- The manuscript (both by the subject of study and the presentation) ends up being targeted towards an audience already quite knowledgeable in the statistical physics approach to the theory of statistical learning, \n  and because of this the NeurIPS audience might not be the best fit for it in my opinion.\n  I tend to think it would be better suited e.g. in venues closer to statistical physics interests.\n  On the other hand, I also understand the upsides of introducing part of this audience to these powerful techniques, so I don't consider this a very weak point.\n- The main weakness of the paper, in my eyes, is the seemingly small novelty with respect to the analysis of \n  [Ghio \\& al '24], which conducts a very similar analysis, although the authors claim that it applies only to planted models (line 59) or normalized densities (line 139). \n  The replica computation seems very similar in nature, modulo the computation of $Z^{-1}$ with an additional use of the replica method. \n  At a high level, this work seems like a fairly close computation, applied to $p$ being the solutions of perceptron problems, rather than e.g. graph coloring or p-spin models which are analyzed in [Ghio\\&al '24].\n\nQUESTIONS:\n- (Line 36): The motivation of sampling from such unnormalized probability distributions is quite brief, and \n  could be developed further.\n\n- (Line 59): I think that [Ghio\\&al '24] also considers sampling from non-planted distributions, such as the p-spin model and the solutions of random graph coloring.\n  Could the authors correct me, or clarify what they mean?\n  On line 139, it is also implied that the specificity of [Ghio\\&al '24] is to consider normalized densities, which makes it a bit confusing.\n  More generally, I would be potentially willing to change my rating and confidence score \n  if the authors were to convince me better regarding the technical novelty of their work with respect to [Ghio\\&al '24].\n\n- I think that at several points (e.g. regarding the introduction of the free entropy line 150, or the replica results line 177), \n  reading the paper might be complex for an audience which is not expert in statistical physics techniques. It would be good to give a more pedagogical presentation \n  of these tools for this audience, which is the vast majority of the NeurIPS attendees.\n\n- In the conclusion, the authors mention that the $\\tau$-annealing procedure proposed overcomes ASL's limitation on structured instances of the problem, but this is not explored in the paper. \n  I imagine this is rather an informal conjecture made by the authors, by analogy with similar statistical learning problems?\n  Moreover, the authors mention that the ideas developed in this paper could enhance \"sampling and solving\" in other hard CSPs: I wonder how would the present analysis help in solving (i.e. finding a solution to the problem in a given instance)?\n\nFinally, I list here more minor points:\n\n- (Line 66) I believe the negative spherical perceptron was introduced before the work of Franz \\& Parisi in 2016. \n  While I don't know the earliest references, it is studied for instance in the work \"Negative spherical perceptron\" (Stojnic '13), \n  and is mentioned in Chapter 8 of the book \"Mean-field models for spin glasses\" (Talagrand '10), see Chapter 8 (for instance in Problem 8.4.3).\n\n- (Line 122) It would be good to mention again in this paragraph the works that introduced ASL, and that are cited in the introduction.\n\n- (Line 128) I would say that in terms of limitations of AMP, time complexity should also be mentioned, it's possibly a stronger limitation in practice than low-dimensionality of the data.\n\n- Equation (12): the introduction of both $T$ and $U$ is a bit redundant, and makes part of the exposition slightly confusing (since $T$ is not really the temperature of the problem, as we are still trying to sample zero-energy states and not the Gibbs measure at temperature $T$). Maybe a different name for $T$ would be best.\n\n- (Line 243) It is quite surprising at first sight that the paratmeters of the reference system are independent of $t, q$ and $\\hat{q}$, since they interact via the replica $w_1^\\star$: could the authors comment on that? \n  Does this suggest some effective decoupling of the term $Z^{-1}$ with the other terms in the replica computation? \n\n- (Line 263) Since the computation here is done assuming that $p$ is RS, wouldn't the whole approach break down beyond the dAT line? \n  As such, can the authors conclude \n  from this calculation that sampling being hard in a RSB phase follows from their analysis?\n\n- Finally, I found the following typos/misprints:\n  - Line 110: I think $p_{\\mathbf{h}, t}(\\mathbf{w})$ is not really a convolution with the Gaussian noise (this would rather be the distribution of $\\mathbf{h}_t$).\n  - (Line 233) The wording \"quite fast\" reads off as a bit vague.\n  - (Line 244): \"system free entropy\"\n  - (Line 246): \"allow us\"\n  - Caption of Fig.2: I would say the agreement \"suggests\" rather than \"shows\" (although \n    this analysis is indeed a convincing argument for the fairness of produced samples).\n  - (Line 285) It should be \"El Alaoui\" rather than \"Alaoui\"\n  - (Line 325) \"for SA\"\n  - (Line 327) \"we sample\" might be a better word than \"we solve\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the use of Algorithmic Stochastic Localization (ASL), a diffusion method combining Stochastic Localization (SL) and Approximate Message Passing (AMP), to sample from target distributions defined over solution spaces of perceptron models. Using the replica method, the authors derive asymptotic thresholds for successful sampling. They apply the framework to two cases: the spherical perceptron with negative margin, where uniform sampling is possible in the replica symmetric phase; and the binary perceptron, where they identify a log-shaped potential enabling efficient sampling and introduce a $\\tau -$annealed MCMC scheme for practical implementation.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nThe theoretical analysis is carefully executed and builds on established tools from statistical physics, notably the replica method. The connection between the shape of the free entropy landscape and the success of the ASL sampling scheme is well described. The application to two well-studied models provides useful benchmarks, and the introduction of a log-shaped potential—motivated by the asymptotic analysis of an unnormalized probability density—yields an efficient sampler over a broader range of $\\alpha$ values for the binary perceptron. Numerical experiments are well-designed and support the theoretical claims.\n\n**Clarity**\n\nThe exposition is clear, with a logical structure and helpful figures. The authors carefully articulate how their work differs from [Ghio et al., 2024].\n\n**Significance**\n\nThe paper addresses a relevant question: under which conditions can generative diffusion methods be used to sample from complex high-dimensional distributions in constraint satisfaction problems. Naturally, the setting is idealized and, although it may be far from practical cases, it provides insights into the interplay between algorithmic tractability and diffusion-based generative models. The proposed annealed MCMC scheme, based on the potential identified in the theoretical analysis, is particularly relevant for binary perceptron sampling.\n\n**Originality**\n\nThe extension of the replica analysis of diffusion-based samplers beyond planted models to unnormalized target distributions is a novel contribution. The identification and use of a diverging potential to enable tractable sampling in the binary perceptron setting, along with the associated annealing scheme, adds originality to the algorithmic aspect of the work.\n\nQUESTIONS:\n**1.** In Figure 1, the value of $\\alpha$ varies around $300$. Is this just a matter of scaling, or does it imply that one needs $M\\sim 10^2 N$ samples in order to reach the regime where AMP fails for the spherical perceptron?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study algorithms in the family of the recently introduced \"algorithmic stochastic localization\", generative diffusion algorithms for sampling where an approximation to the score function is not learned from data but is implemented directly using knowledge of the structure of the target distribution. In particular, as in previous work on the Sherrington-Kirkpatrick model and related applications, the authors use a variant of approximate message passing (AMP) to approximately compute the score function. The authors extend some of the prior work in this area to problems without a statistical structure of a \"hidden\" or \"planted\" signal to be estimated, but rather \"pure optimization\" problems where one seeks to sample something like solutions of a constraint satisfaction problem (though in a continuous setting). They develop tools for applying the (non-rigorous) replica trick of statistical physics to calculate when such sampling algorithms succeed. As applications, they study sampling problems associated to the spherical and binary perceptrons. In the case of the binary perceptron, they show that while the uniform distribution cannot be sampled in this way, a certain tilting of it motivated by the replica analysis can be sampled.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The replica analysis involves what seems to me to be a significant generalization of the analysis in the recent work (Ghio et al, 2024), which focused on \"planted\" models with a distinguished signal object. While the calculations are technical and, as usual, mysterious-looking, the authors make a reasonable effort to summarize the main highlights of the calculation. The perceptron examples are very helpful for explaining what kinds of conclusions one can draw from this analysis, and the case of the binary perceptron and the potential function designed based on the replica analysis are in my opinion a very nice application. (Without this observation, I would be more tempted to complain that these are quite niche applications.) The numerical experiments associated to this make a convincing case that this analysis leads to a powerful method that outperforms a straightforward implementation of simulated annealing.\n\nWeaknesses: I do not have many complaints about the paper - it convincingly solves the problems it sets out to solve and presents the findings reasonably well. I think it is unfortunately pretty much unavoidable that the discussion of the physics analysis will be inscrutable to some (perhaps many) readers, but, as I mentioned, the authors make a commendable effort to compress the main points into a few pages at the beginning. An intrinsic limitation of this entire approach is that it is mathematically non-rigorous and can only be validated numerically, but the authors acknowledge this and their numerical experiments are thorough and convincing.\n\nQUESTIONS:\n- I appreciate that the choice of $U_{\\tau}$ in (20) is charmingly motivated by the way that the actual replica trick works, and this formal idea is delightful, but that aside it does seem like various other approximations of the logarithm might be just as suitable. Have the authors tried any alternatives to this choice and can they comment on whether or why this is the best choice?\n\n- Could the authors comment on, in terms of the actual distribution of solutions to the binary perceptron problem, what the distribution tilted by their potential looks like compared to the uniform distribution? Which parts of the solution space are favored? Is there any very down-to-earth explanation of why might plausibly make it easier to sample?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}