Title: Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Restricted Boltzmann Machine

Approximate Message Passing

High-dimensional Limit

Multi-index Models
Abstract: The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization.  This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBMs on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning.  We show in particular that RBMs reach the optimal computational weak recovery threshold, aligning with the Baik-Ben Arous-Péché (BBP) transition, in the spiked covariance model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers Restricted Boltzmann Machines (RBMs) in the classic high-dimensional limit of growing input dimension and a growing number of data points (asymptotically proportional) and contributes to the analysis of RBMs in this setting. First, a simplified form of the log-likelihood function is derived in this asymptotic setting (Theorem 1), complemented by corresponding forms of the gradient and some classic further simplifications. This is then used together with the classic Spiked Covariance Model to analyse Approximate Message Passing (AMP) on the RBM (which in turn forms a new approach to train RBMs), in particular, the gradient of the simplified log-likelihood at the AMP fixed point (Theorem 2) and the convergence behaviour of AMP under certain initializations (Theorem 3). Furthermore, again using AMP, an asymptotic characterization of stationary points of the log-likelihood are provided (in Theorem 4) and a result on the convergence behaviour of gradient descent (Theorem 5), which is complemented by numerical experiments on synthetic data and Fashion MNIST. The main results are complemented by generalizations (e.g., a more general AMP algorithm) and a result on the global optimum and the AMP fixed point for a simplified variant of the Spiked Covariance Model under standard assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nThis work contains a thorough theoretical investigation of RBMs in a classic asymptotic setting, with a particular focus on AMP and gradient descent, both under the Spiked Covariance Model. This is a relevant and active area, and the aim of the authors to provide a rigorous analysis (instead of using heuristics) makes the work particularly relevant. Furthermore, the proposed AMP algorithm in the asymptotic setting might be of independent interest as a new training approach to RBMs.\n\nAs clearly stated in the manuscript, using AMP to analyse RBMs is not new, and essentially all of the techniques used are established (though might be relatively recent, for example the developments in Section D), though making everything rigorous would still constitute a very relevant contribution.\n\nUnfortunately, the work suffers from many missing details and imprecisions, cf. the detailed comments and suggestions below and the questions, in addition to typos and some language problems. Since the explicitly stated goal of the present work is to provide a rigorous asymptotic analysis, the imprecisions and missing details are a major problem, and indeed quite a few steps and results were difficult to check for the present reviewer. If these problems are addressed and resolved (which would require a careful revision of the manuscript), this work can be a very interesting and promising contribution to the analysis of RBMs, with potentially considerable follow up work.\n\n**Detailed comments and suggestions**  \nL10\nLanguage, \"via the dynamical mean-field theory\" is probably more common\n\nL11, 13\nLanguage, \"RBMs\" is probably better in terms of grammar\n\nL14\nAbbreviation BBP not introduced (in contrast to all other abbreviations in the abstract)\n\nL20\nPotentially ambigous sentence: Not state of the art for image generation anymore, but for other tasks? Made add an \"for example\"\n\nL36\nGrammar, \"yields\"\n\nL38\nLanguage, a connecting word is missing\n\nL59\nUnclear, does \"mathematically\" refer her to general analytical techniques (including formal derivations), or rigorous results?\n\nL64\n\"BBP threshold\" has not been introduced yet. In fact, the first reference regarding this is given only in L249.\n\nL107\nTypo, \"Spiked\"\n\nSection 2\nThere are some imprecisions / missing details\n* What are $\\theta$ and $\\beta$? These quantities are never introduced\n* How exactly is the data $X$ used? After its introduction in L113 it is never explicitly mentioned anymore. I guess it enters (3) via $x_\\mu$, the rows of $X$ according to the convention at the end of the section.\n\nL113\nSomething is wrong here (superfluous \"using\"?)\n\nLL115\nSomething is wrong here. $W$, $\\theta$, $b$ appear to be trainable parameters, $v$ and $h$ are data ($v$ visible, $h$ hidden)\n\nL118\nWhy is \"prior\" in quotes? \n\nL118\nMaybe add \", respectively.\" at the end for clarity\n\nEqu (3)\n* Is $Z^n(W,\\theta,b)$ actually the quantity from (2) raised to the $n$-th power? If so, maybe use $Z(W,\\theta,b)^n$ instead to avoid confusion.\n* I strongly suggest to use $h$ instead of $h_\\mu$ since this is a free variable (it is integrated over), so using an index can lead to confusion.\n\nEqu (3)\nMaybe use a different symbol than $\\mu$ for the running index. Since you work a lot with probability distributions, and $\\mu$ is one of the standard symbols for probability measures, this might lead to confusion.\n\nL124-126\nMaybe include examples to illustrate these conventions.\nFurthermore, I strongly recommend to introduce these conventions before they are used for example in Equ (3).\n\nAssumption 1\n* Is separability an established terminology in the present context? Because technically, this just means that the random variables $v_1,\\ldots,v_d$, where $v \\sim P_v$, are independent (the density w.r.t. Lebesgue measure of their law factorizes)\n* What is meant by \"a bounded distribution\"? Bounded support? The density is a bounded function? Or do you mean by \"distribution\" any density w.r.t. Lebesgue measure, leading to a (nonnegative) measure that is not necessarily a probability measure?\n\nEqu (6)\nIncompatible dimensions (wrong use of transpose). The first argument of $\\eta_1$ in (5) needs $W^\\top x_\\mu$, not $x_\\mu^\\top W$\n\nL147\nInconsistent use of commas, \"i.e.\" and \"i.e.,\", \"e.g.\" and \"e.g.,\". The remainder of the manuscript should also be checked for this issue.\n\nL148\nGrammar, missing article\n\nEqu (8)\nWhere exactly is the proof of this result?\n\nL154-157\nA reference should be given for this.\n\nEqu (9), (10)\nThe redefinition of the symbols on the lefthand side in these equations is potentially confusing, maybe choose different symbols.\n\nL155, 157\nWhile the notation $\\langle\\cdot\\rangle$ is very common in statistical physics, it rarely appears in statistics and (theory of) machine learning, so it should be briefly introduced somewhere.\n\nL181\nWording, something is wrong here. Maybe \"slightly generalized\"?\n\nEqu (12) \nWhich matrix norm is used here?\n\nL187, \"then we have\"\nBetter to use $\\approx$ instead of $=$\n\nL197\nWording. Maybe \"the teacher-student setting\"?\n\nL204\nTypo, duplication of \"now\"\n\nL204\nThis can be made a bit more precise, e.g., \"following the spiked covariance model (11)\".\n\nEqu (15)\nTechnically, $\\eta_1$ and $\\eta_2$ with just one argument are not defined (I guess it corresponds to $\\eta_1,\\eta_2$ from Theorem 1 with the remaining arguments set to zero)\n\nEqu (15)\nMinor remark: Optimization problems are usually formulated with $\\max$, and $\\mathrm{argmax}$ is only used when one defines a solution to it.\n\nL205\nWording, \"direct\" is not appropriate here. Similarly in L237\n\nLL206, \"While it is common to use AMP for supervised generalized linear regression\"\nReference?\n\nL209\n* Grammar, use singular \"aspect is\", or \"Other new aspects\"\n* Typo, \"penalty\"\n\nL210\nTechnically, the following display is not the new loss, but rather an optimization problem with the new loss.\n\nLL212\nConfusing. Is Algorithm 1 already \"AMP-RBM\", or is the latter a specific instance of the former?\n\nAlgorithm 1\nOn which AMP variant is Algorithm 1 based? It is not the one from Montanari and Venkataramanan, 2021 (e.g., there is an additional \"damping term\" here)\n\nL213\nGrammar, \"the denoisers\"\n\nL219\nGrammar, missing \"that\"\n\nL220\nWording, better to use \"Let ... be the\"\n\nFigure 1, caption\nTypo, \"initialization\"\n\nL228\nThe notation $U_{[1:r]}$, $W_{[1:r]}$ is not defined. Furthermore, what does $[G]_1^k$ mean?\n\nTheorem 3 / Footnote 1, \"Lipschitz continuous, uniformly at\"\nThis terminology is not quite standard, so it might be better to defined it explicitly in Theorem 3.\n\nLL244\nA reference regarding the weak recovery threshold should be added here.\n\nLL251, \"AMP-RBM as well as RBM (because their fixed points match)\"\nWhere exactly does this follow from?\n\nL275\nInconsistent use of cases, \"Assumptions\".\nSame issue in L287.\n\nCorollary 1\nWhat does \"can be diagonal\" exactly mean?\n\nL282\nTypo, \"techniques\"\n\nL289\nIt would be nice to provide a pointer to Section C.5 with the proof of Corollary 1 for the reader's convenience.\n\nL303\nGrammar, \"preferred to\"\n\nEqu (25)\nHow exactly is this equation related to what came before?\n\nL317, \"converges to $\\hat{W}$ uniformly,\"\nUniformly w.r.t. what?\n\nL319\nWording, better to use something like \"corroborated\" or \"illustrated\" instead of \"verified\"\n\nReferences\nThe reference entries should be carefully checked, upper / lower case use is inconsistent with the main text (e.g., boltzmann instead of Boltzmann in Salakhutdinov et al '07)\n\nL502 \nThe inverse temperature $\\beta$ is not introduced yet, the paragraph needs to be adapted accordingly.\n\nL503\nMissing comma\n\nL505\nWhat is $\\beta^\\ast$? I guess it refers to L192\n\nL512\nMissing comma\n\nL512\n\"maximizes... instead\" Reference / derivation?\n\nL514, \"impossible for inference\"\nLanguage, something is wrong here\n\nL514-516\nMaybe refer to the location of this result in the main text\n\nL519\nMissing comma, or use \"the following\"\n\nL522\nWhere are the arguments $\\theta$, $b$?\n\nL523, \"the derivative of the second term\"\nSecond term of what?\n\nEqu (35)\nThis uses the separability from Assumption 1. This should be made explicit.\n\nEqu (36)\n$\\mathcal{Z}$ not defined\nHow does the last line follow?\n\nL529, \"(36) also directly proves (4).\"\nHow? (4) does not involve any derivative.\n\nL535\nGrammar and Language\n\nL536, \"The gradients can be dealt with in a similar way\"\nWhat exactly does this mean?\n\nL547\nMissing comma\n\nL552\nTypo\n\nAssumption 3\nIs this assumption new? Or has it been used in the literature before? It is not clear what the relation to the references in LL560 is.\n\nL555\nWhat is $\\|\\mathcal{F''}\\|$? Supremum norm? Measurable essential supremum?\n\nEqu (5)\nInconsistent / undefined notation. I guess $\\mathcal{F}^{1}=\\mathcal{F}'$.\n\nL557\nWording. Better to write something like \"i.e., the data are well-normalized\"\n\nL559\nLanguage\n\nL564\nWhat exactly is meant by \"normalized\" here? That the matrix is normalized?\n\nL564\nWhat is \"the error matrix\"?\n\nEqu (52)\nThere is an imprecision here. In general, $n/d\\not=\\alpha$, only $n/d\\rightarrow\\alpha$, cf. also the comments and questions regarding Theorem 1.\n\nL570\nHow exactly does $\\mathbb{E}[\\tilde Z_{ij}]=1$ follow from Assumption 3 (presumably last part of (ii))?\n\nL572\nInappropriate notation. I guess $L^2$ refers to the supremum norm (the $\\ell_2$ norm for a matrix space would be the Frobenius norm).\n\nSection C.2\n* The AMP variant outlined here is not the one from Montanari and Venkataramanan, 2021, Supplement Section J, since here the full history of the iterates are considered in the denoising functions. On which AMP scheme is this section based? It seems to be (a variant of) Zhong et al '24, Section 2.2\n* Inconsistent notation for Onsager coefficients (subscript with and without comma)\n\nEqu (56) and Equ (57)\n$F^t$, $E^t$ in (56) and $F$, $E$ in (57 are not defined. I guess these are generic side information, cf. Zhong et al '24, Section 2.2\n\nTheorem 6\n* The AMP variant and the setup is sufficiently different from Zhong et al '24 to require a full proof.\n* The wording of this result could be improved.\n\nL592\nGrammar, use plural\n\nL604\nTypo, \"Lipschitz\"\n\nEqu (64)\nSome details on how to derive the second equation should be given. I guess the argument is as follows: By definition, $h_t$ is a stationary point of the objective function in the minimization in (17), so the gradient of this objective function at $h_t$ is zero, and it can be rearranged for the latter, and in can be plugged in the update rule for $U^t$ in Algorithm 1. Finally, the update rule can be rearranged for $h_t$, and this expression is plugged in as the argument for the derivative of $\\eta_1$.\n\nL614, \"Moreover, at the fixed point we have...\"\nHow? I guess it follows from the last line of Algorithm 1.\n\nL625, \"which suggests that the limit in (73) exists\"\nImprecise. Does the preceding constitute a rigorous proof or not?\nSimilar issue in L627\n\nEqu (74)\nMissing closing bracket\n\nEqu (76)\nSuperfluous opening bracket. Furthermore, the multiplicative factor needs to be $\\alpha^{-1}$\n\nL653, \"as the following\"\nWording, I guess what is meant is \"as follows\"\n\nL641\nWording\n\nL673-679\nPutting this part into an Example environment would improve readability.\n\nL681, \"we need to give some notations\"\nWording, better to use something like \"we need to introduce some notation\"\n\nL696, \"Now we present our main results,\"\nProblematic formulation since this is part of the supplementary part and only briefly mentioned in the main text. Maybe use \"main results in this section\" or something similar.\n\nEqu (96)\nWhat is $A$? I guess it is supposed to be $\\mathcal{A}$, similar to $\\mathcal{A}^\\ast$ in (8) in Viluchio et al '25\n\nLL712\nWhat exactly is meant by uniformly here?\n\n(102), second line\nSomething is wrong here. I guess in the second term in the sum a $w$ is missing (and indeed it appears in (103))\n\nL726\nWhy can Lemma 4 be used here? It requires minimization / maximization over compact sets.\n\nL727\nWhat is the resulting direction of $f$, and how does (104) then follow from it? More precisely, how does the second term in the first line of (104) follow in this way?\n\n(105)\n$\\phi$ is not included in the min / max anymore\n\nL734-736\nHow exactly does this follow from the (strong) law of large numbers?\n\nL740 Typo\n\nLL753\nHow exactly does this follow?\n\nLL793\nTerminology, \"contrastive divergence\"\n\n(137)\nProblematic notation. $\\zeta_{ij}$ is already defined, (137) is just an estimation based on SE. Maybe use $\\hat\\zeta_{ij}$ instead.\n\nL827\nTypo, \"realistic\"\n\nFigure 4\nWhat are the \"error bars\"? Standard deviation?\n\nLL835, \"AMP-RBM might fail to converge for k= 10,\"\nWhy?\n\nL851, \"corresponding to Figure 5\"\nI guess it means something like \"with the same setting\" or \"analogously\"\n\nL858, \"decimation AMP\"\nWhat exactly is this?\n\nQUESTIONS:\n**Question 1** Related to Theorem 1  \nThe following aspects are relevant for *Clarity*, *Quality*, *Significance*, and *Rating*\n\nQ1.1 Lack of precision  \nFor example, Equ (4) can only hold if \n$d,n\\rightarrow\\infty$ simultaneously with \n$n/d\\rightarrow \\alpha$, cf. L123. \nI guess the setup is the following. \nLet $(n_l)_l$ and $(d_l)_l$  be two nondecreasing, unbounded sequences of integers such that \n$\\frac{n_l}{d_l} \\rightarrow \\alpha$ for $l\\rightarrow\\infty$. Furthermore, let $X(l)\\in\\mathbb{R}^{n_l \\times d_l},\\ldots,\\theta(l)\\in\\mathbb{R}^{d_l},b(l)\\in\\mathbb{R}^k$ \nbe such that the conditions in L143 hold. Then for $l\\rightarrow \\infty$, we have\n$\\frac{1}{d_l}|\\log \\mathcal{L}(W(l),\\theta(l),b(l)) - \\log\\tilde{\\mathcal{L}}(W(l),\\theta(l),b(l))| \\rightarrow 0$.\n\nThat this is the intended setup is indicated by LL142, where $b(n)$ and $b(d)$ appear (this is also confusing due to two different parametrizations).\n\nA similar issue appears in (20), (23), (24), (29), (30), Lemma 1 (in (52), cf. the corresponding comment), and (61).\n\nQ1.2 Proof strategy for Theorem 1  \nIn L519, it is stated that \"For the following we will prove a stronger result\". But how does Theorem 1 follow from (32)? In particular, (32) involves the gradient of the likelihood, whereas (4) concerns the likelihood itself. The connection between the two is not clear at all.\n\nFurthermore, the term involving $\\eta_1$ in (5) seems plausible (after doing a brief, elementary calculation), but it is not clear at all how the form of $\\eta_2$ appears.\n\nQ1.3 Equ (39)  \nHow exactly does it follow from (38)? It seems to be a rather lengthy computation.\nSimilarly for equ (40) and (41).\n\nQ1.4 L523, \"The first term is exactly $\\eta_1$\"  \nThis does not seem to be the case. $\\alpha$ is defined as the limit of $n/d$ for $n,d\\rightarrow\\infty$, but in (33) (using $\\frac{1}{\\sqrt{d}}=\\sqrt{\\frac{n}{d}}\\frac{1}{\\sqrt{n}}$) only $n/d$ appears, but not its limit. Furthermore, (33) considers the likelihood from (3), not its gradient, so how does this help to show (32)?\n\nQ1.5 Equ (34)  \n* First line: What happened to $P_h(h)$ from (33)?\n* Second line: Where is the second term in the sum inside the exponential inside the integral?\n* Third line: How does this follow?\n* Fifth line: How does this follow? Where does the exponential come from?\n\nQ1.6 Equ (35)  \nWhere does the $P_v$ in (35) come from? It has been integrated out in (34)\n\nQ1.7 Equ (38)  \nHow does the second equality follow? Where does the exponential come from?\n\n**Question 2** Related to Theorem 2  \nThe following aspects are relevant for *Clarity* and *Quality*\n\nQ2.1 Equ (17)  \nHow is it ensured that $B_t^{-1}$ is invertible?\nDoes the minimum in the definition $\\tilde{h}^t$ exist? Is it unique? If it is not unique, which minimizer is chosen?\n\nQ2.2 Assumptions  \n$\\eta_1$ and $\\eta_2$ are defined in Theorem 1 analytically, so these functions are either continuously differentiable or not. So are they continuously differentiable or not? And where exactly is this used in the proof of Theorem 2? It seems that only their differentiability is used (to define $\\nabla \\eta_1$, $\\nabla \\eta_2$).\n\nQ2.3 Theorem 2, \"at the fixed point\"  \nDoes a fixed point always exist, and if so, is it unique? This is not discussed, and no reference is given.\n\nQ2.4 Equ (18)  \nWhere does $\\alpha$ come from? Using the definition of $Q(W)$ from L153, there should be $1/d$ instead.\n\n**Question 3** Related to Theorem 3  \nThe following aspects are relevant for *Clarity*, *Quality* and *Significance*\n\nQ3.1 Choice of $\\psi$  \nWhat would be an interesting PL(2) function $\\psi$ in this context? \nIs the choice in L279 in the context of Theorem 4 the only relevant one?\n\nQ3.2 Limit notion  \nAnd what can be inferred from the Cesaro convergence (26)? In other words, which statements about the behaviour of the RBM in the asymptotic setting can be given using this limit notion?\n\nQ3.3 Extension  \nDoes a similar result like Theorem 3 hold also for the $u_i^\\ast$?\n\n**Question 4** Additional questions related to (significant) technical details  \nThe following aspects are relevant for *Clarity*, *Quality* and *Rating*\n\nQ4.1 Equ (3)  \nWhere is the $dP_v(v)$ term? Is it just assumed that it equals 1? If not, which adjustments are needed for the remaining developments?\n\nQ4.2 LL209  \n\"We deal with the term $\\eta_2$ by viewing it as a set of constraints and introducing Lagrange multipliers to fix them\"\nThis is unclear and requires further elaboration. In particular, $\\eta_2$ is just an additive term in the objective function of the optimization problem, so in which sense is it a constraint?\nFurthermore, What is meant by \"to fix them\"? In general, a Lagrangian formulation is a relaxation, and one needs conditions to enforce equivalence.\n\nQ4.3 Equ (73)  \nHow does this follow from Theorem 3? \n\nQ4.4 LL627  \nHow exactly does the zero convergence stated in line L627 follow from Theorem 2? The latter does not involve the diverging factor $\\sqrt{d}$, and since Theorem 2 does not provide any convergence rate, it is not clear that the convergence is fast enough to overcome the growth of $\\sqrt{d}$.\n\nQ4.5 Equ (82)  \n* L653, \"Then it suffices to map (81) to a general AMP iteration as the following\"\n\tWhich general AMP iteration? The one in (56), (57)?\n* Theorem 5 uses $t\\geq 0$, but some subexpressions are in (82) are not defined for $t=0$.\n* Something seems to be off with the mapping to the general AMP iteration. (25) can indeed be rewritten as (81), however, it is not clear how to get from (81) to (82). In particular, when plugging in the various quantities in (82), $\\hat{W}^t$, $\\overset{\\circ}{W^t}$ etc get mixed up. More details need to be provided on how to perform this mapping.\n\nLIMITATIONS:\nIn general yes. However, while the setting used is classic, its inherent limitations should be briefly stated.\n* Apart from Theorem 1, mostly a synthetic data setup is considered, which might be unrealistic from a practical perspective.\n* This work considers a particular asymptotic setting. While interesting, it is still an approximation, and since no convergence rates are given, it is unclear when this approximation can be considered sufficiently accurate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers RBM in the asymptotic limit where the input dimension goes to infinity but the number of hidden units remain finite, and proves that the RBM training objective is equivalent to that of an unsupervised multi-index model with a non-separable penalty in this regime. The paper then proceeds to apply and extend established mathematical techniques for the multi-index model to analyze the asymptotic behavior of RBM trained on the spiked covariance data, and derived closed-form equations describing the gradient-descent dynamics of RBM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper presents theoretical results on the sharp asymptotics of RBM. The connection established between the RBM and the unsupervised multi-index model is general and does not rely on specific data assumptions, this connection enables simplified approximations of the log-likelihood and its gradient, and could potentially be used to accelerate training, as the authors pointed out. For more precise theoretical calculations, the authors then model the inputs as a spiked covariance model, and derived the AMP iterations and its corresponding deterministic SE, and derived the asymptotics of GD dynamics. Combined with thorough numerical validation and empirical results, this paper provides rigorous theoretical understanding of the learning dynamics of RBM. The paper is also clear and well-written. \nOverall, while the paper is still limited by its data assumptions (see weaknesses), it presents elegant and accurate theoretical results for the RBM and verified them numerically, extending mathematical results in supervised learning to a type of unsupervised generative model. \n\nWeaknesses:\nAs mentioned earlier, the main theoretical result of this paper relies on the data assumption, i.e., the spiked covariance data model, which is a relatively simple model for unsupervised learning. The authors did not discuss how their results may provide insights for data beyond this restrictive assumption. I believe the paper would largely benefit from some numerical experiments showing that behavior on some benchmark datasets for unsupervised learning can be also (perhaps qualitatively) captured by the theory. For example, the authors already showed results verifying theorem 1 in the appendix using Fashion-MNIST, is it possible to approximate Fashion-MNIST using a spiked-covariance model and show comparison between AMP-RBM and GD on this data?  \nAlso, the main technical contribution of this paper is 1) proposing the analogy between RBM and the multi-index model in the asymptotic limit 2) deriving the AMP equations and analyzing GD dynamics of RBM in the asymptotic limit and under the spiked-covariance data assumption. While the first point is novel and insightful, the second point largely build on existing work on multi-index models, and the major theoretical advancement is the extension to coupled regularization. The extension itself is relatively straightforward technically as the number of hidden units is finite. While this is not necessarily a weakness, I do think that because the technical advancement is relatively straightforward, there should be more emphasis on how the new theoretical result on RBM can provide new insights. For example, the paper would benefit from a discussion highlighting properties predicted by the RBM theory that are not in the multi-index model. \nThe paper could also benefit from a conclusion/summary section to discuss the potential impact and possible extensions of the work.\n\nQUESTIONS:\n1. The paper suggests around line 80-84 that statistical physics approaches is closer to Bayesian sampling and considers the full posterior. However, isn't their approaches equivalent to maximum likelihood when taking the temperature to 0 limit?\n2. In section 4, when comparing the spiked covariance model to an RBM teacher, you need to assume U is binary and W is approximately orthogonal, these are additional assumptions on the spiked covariance model that you don’t need for the following theoretical results, is that correct? If my understanding is correct, maybe you should be more explicit about it in this section.\n3. Some numerical results are present in the Appendix but not explicitly referred to in the main text. For example, when discussing the convergence of different AMP algorithms in line 295-299, it is maybe useful to point to Appendix F.1 where you showed numerical results comparing the performance of AMP with random initializations or more informed initialization schemes. \n4. What are the actual distributions P_u and P_w used in your numerics? Do their forms affect the convergence of AMP-RBM?\n5. I am a bit confused about the results showing overlap vs. SNR (or lambda) in the main text vs. the appendix. Just to confirm my understanding, in the main text you showed results with r=k=2, and the lines are SE corresponding to Eq. 19, which should hold irrespective of Theorem 4 and the k=r=1 assumption. In the appendix you stated that the lines are for rank-1, which I suppose refer to k=r=1, where the line now corresponds to Theorem 4, which assumes the convergence of AMP-RBM, is this correct? What would SE look like for k=r=10?\n6. This may be a broad and less relevant question. This paper proposes potential simplification of the gradient of the log-likelihood in the asymptotic limit, could you perhaps discuss whether the results may provide any insights to score-based diffusion models, a very different type of generative model that is more commonly used in recent applications, and also considers approximating the gradient of the log-likelihood of data?\nAnd some typos...\n7. Line 115, the biases should be theta and b? \n8. Line 117, isn’t the energy term order 1 after adding the 1/sqrt(d) normalization, so should be ‘intensive’ instead of ‘extensive’? \n9. In the appendix, there should no longer be integrating over v in the second line of Eq 35.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies RBM in the limit where the input space has large dimensionality while the number of hidden units remains fixed. The authors connect RBM training objective to a form equivalent to a multi-index model with non-separable regularization. This reduction enables them to use established analytical tools from the study of multi-index models—such as Approximate Message Passing (AMP) and its state evolution, as well as the analysis of Gradient Descent (GD) via dynamical mean-field theory—to investigate RBM training dynamics. They derive rigorous asymptotic results for RBM training on data generated by the spiked covariance model, which serves as a prototypical example of structure amenable to unsupervised learning. In particular, they show that the RBM achieves the optimal computational weak recovery threshold, corresponding to the BBP transition in the spiked model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Establishing the connection with multi-index model, and applying AMP to study RBM in the asymptotic limit of large dimensionality, obtaining asymptotic results for data generate by spiked covariance model.\n\nWeaknesses: the part of the analysis of GD is weak. The asymptotic DMFT eqs are derived and proven but no analysis, even a heuristic one, is provided. The work of Bachtis et al, and Harsch et al provide such analysis in a non-rigorous setting.\n\nQUESTIONS:\nIf possible, it would be interesting to provide more results on the GD part: (i) can the authors provide at least heuristic arguments for the the weak recovery threshold of GD backing up the empirical results of Fig (1) ? (ii) can the authors provide an analysis of the GD dynamics  ? \n\nIn \"related works\" the discussion of Bachtis et al, and Harsch et al should be improved. These works provided a heuristic analysis of the GD dynamics as a function of training time - the sentence \"Their analysis is non-rigorous, whereas our results provide exact asymptotics, including the characterization of saddle points\" seems to imply that this work contains a more complete analysis of GD dynamics, which is not the case.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}