Title: Energy-based generator matching: A neural sampler for general state space
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural sampler

Boltzmann distributions

diffusion model

flow model
Abstract: We propose Energy-based generator matching (EGM), a modality-agnostic approach to train generative models from energy functions in the absence of data. Extending the recently proposed generator matching, EGM enables training of arbitrary continuous-time Markov processes, e.g., diffusion, flow, and jump, and can generate data from continuous, discrete, and a mixture of two modalities. To this end, we propose estimating the generator matching loss using self-normalized importance sampling with an additional bootstrapping trick to reduce variance in the importance weight. We validate EGM on both discrete and multimodal tasks up to 100 and 20 dimensions, respectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of drawing independent samples from an unnormalised target distribution $p_{\\rm target}(x)\\propto \\exp(-E(x))$ when only an energy function E(x) is available. Building on the recent generator matching (GM) framework, which trains continuous-time Markov processes (CTMPs) to follow a prescribed probability path via minimisation of a Bregman divergence between marginal and conditional generators, the authors introduce and energy-driven training of CTMPs by estimating the marginal generator via self-normalised importance sampling (SNIS).  Under this approach, a proposal distribution $q_{1|t}$ is used to draw candidate endpoints, and weights are computed from the energy function to approximate the intractable posterior over data endpoints. One of the challenges with the naive implementation of this scheme is that the variance in the importance weights can be high. To alleviate this issue, the authors introduce a bootstrapping approach to reduce the high variance of importance weights by introducing an intermediate at time $r>t$ proposal. The paper includes an empirical study on synthetic benchmarks up to 100 dimensions (Ising grids at various temperatures) and joint discrete–continuous tasks (GB-RBM, double-well sequence–structure co-generation, mixture-of-Gaussians). In all cases, the proposed EGM scheme appears to outperform a parallel Gibbs baseline in terms of energy-Wasserstein distance.\n\nSTRENGTHS AND WEAKNESSES:\n$\\underline{\\text{Quality}}$\n\nThe paper is methodologically and theoretically sound. The authors present very clear derivations of all of the important concepts in the paper, e.g. importance sampling, generator matching, etc. without any obvious omissions. The paper is very well-structure and easy to read. The method itself is clearly described in the algorithm box, making it easy to recreate the methodology. The empirical study is convincing, in terms of how the method compares to the Gibbs sampler approach. \n\nHowever, there are a few weaknesses that might be possible to address. The numerics are a bit simple. The paper omits any systematic analysis of computational complexity. There is no clear guidance on the number of energy evaluations required, the wall-clock time per training epoch, or the memory footprint relative to baselines like Gibbs sampling or diffusion-based methods. There is also a lack of comparisons to other methods, e.g. generator matching (with the exception of Figure 1). Additionally, while variance reduction via self-normalised importance sampling does seem to offer an improvement, the bias introduced by this scheme, and also the surrogate energy approximation, is only discussed informally. Currently, there are no theoretical bias bounds, nor does the empirical study show how bias scales with the number of importance samples, or how surrogate accuracy impacts the results. Finally, all empirical validation is confined to synthetic tasks in at most 100 dimensions, leaving the method’s performance on large-scale, high-dimensional, or real-world problems unknown. \n\n$\\underline{\\text{Clarity}}$\n\nThroughout the manuscript, notation is clear and terms such as: variables, generators, and kernels are introduced in a systematic fashion. The notation also follows a similar style to other papers in this area, which helps readers to compare this paper to other similar works. The figures, especially Figure 1, are well-presented and help to understand the methodology and its relationship with other approaches. The pseudocode in Algorithm 1 offers a clear, step-by-step recipe that helps with comprehension and implementation. \n\nOverall, the paper is very clear, and I think readers who have read the related papers, e.g. flow matching, will find it easy to follow this paper. However, some of the notation occasionally is a bit complicated (but probably hard to avoid) e.g.  symbols such as $\\tilde p$, w, and multiple kernels appear in rapid succession, forcing readers to frequently backtrack to distinguish unnormalised densities from importance weights. Furthermore, some critical results, especially those concerning the intractable backward kernels and the precise role of the surrogate energy, are relegated almost entirely to the appendices.\n\n$\\underline{\\text{Significance}}$\n\nA key contribution of this work is its unified treatment of diverse data modalities. By casting EGM in a measure-theoretic framework, the authors demonstrate how continuous, discrete, and hybrid sampling problems can all be addressed under a single continuous-time Markov process paradigm. This is a nice contribution which addresses a gap in energy-based generative modelling. The bootstrapping trick, which introduces a learned surrogate energy at an intermediate time for variance reduction, represents an interesting approach that has potential utility beyond the specific context of EGM, suggesting broad applicability to other importance-sampling and sequential Monte Carlo methods. Moreover, the method directly addresses scenarios in which only an unnormalised energy is available and no pre-sampled data exist, offering practical value in scientific and engineering settings where normalisation constants cannot always be computed.\n\nHowever, the true significance remains to be demonstrated in large-scale, applied domains. The current empirical evidence is restricted to synthetic benchmarks, leaving open the question of whether the theoretical benefits translate to real-world gains in fields such as molecule generation or image generation. The empirical comparisons are also narrowly focused on parallel Gibbs sampling; without head-to-head evaluations against cutting-edge alternatives. For example, it's a bit surprising that there are no comparisons with any diffusion-based schemes or flow matching, or even generator matching (especially given that this paper builds on generator matching). Therefore, the claim of broad applicability is weakened. Finally, while the unification of modalities is appealing, some may view the contribution as an incremental extension of existing generator-matching, rather than a unique and novel contribution.\n\n$\\underline{\\text{Originality}}$\n\nEmbedding importance sampling directly into generator matching is a new idea and an interesting methodological contribution. Although generator matching and diffusion-based samplers have been explored previously, to the best of my knowledge, this paper is the first to integrate self-normalised importance sampling in the marginal-generator estimation step, supported by a clear theoretical justification. The introduction of a bootstrapping mechanism via a learned surrogate energy at an intermediate time is quite creative, offering a new approach to control the variance, which I haven't seen before in the energy-based sampling literature. \n\nThat said, the work does draw heavily on existing frameworks. The mathematical foundation closely parallels that of iDEM and other diffusion samplers, and the bootstrapping idea, while inventive, relies on surrogate modelling techniques. This is potentially an issue because this now adds an additional layer of approximation which wouldn't be needed if the importance proposal used the simpler $q \\propto p$ form. The discussion of connections to GFlowNets and discrete flow models, though present, is somewhat limited (e.g. no numerical comparisons). It would have been interesting to see a deeper exploration of these relationships which could have further highlighted the paper’s distinctive contributions. Ultimately, the originality hinges on the effective combination of the importance sampling, with bootstrapping, within generator matching. This is interesting in itself, but could be seen by some people as a specialised rather than a broadly transformative advance.\n\nQUESTIONS:\n1) Choice of $r$: It would seem that, based on Section 3.2, the method works well when $r$ and $t$ are close, and therefore their densities are similar. However, for the experiments I do not believe $r$ (or rather $\\epsilon$) was stated in the main paper. It would be interesting to test the method's efficacy as a function of $\\epsilon$. \n\n2) Comparisons to generator matching: I was quite surprised that the paper does not compare against generator matching, flow matching, or any other method (with the exception of Gibbs sampling). Can the authors explain why they haven't compared against other than Gibbs sampling? \n\n3) Real-world examples: Can the authors explain why they haven't used a real-world datasets within their empirical study?\n\nLIMITATIONS:\nThe authors do not really address the limitations of their method. Had they explored more comparisons to alternative approaches, this might have led them to better understand the settings under which this method may not perform as well as competing approaches. \n\nThe authors do not discuss the negative societal impacts of their work. However, for this type of paper, I do not believe that this a significant issue.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to solve the problem of sampling from distribution with given energy function (unnormalized log probability). Given a target distribution, the authors introduce a series of intermediate distributions similar to diffusion/flow matching model. They then introduce a generator function $F_{t}^{\\theta}$, which is trained with the generator matching loss. To reduce the variance in the importance weight, the authors further introduce an additional bootstrapping trick. The authors evaluate their method on several toy examples and compared with the basic Gibbs Sampler.\n\nSTRENGTHS AND WEAKNESSES:\n***Strength:***\n\nThis paper proposes a valid sampler and provides sufficient derivations and theoretical justifications to facilitate reader understanding.\n\n***Weakness***\n\nThe proposed method is only evaluated on relatively simple distributions and is compared solely with the basic Gibbs sampler. In fact, the task of sampling from a given energy-function remind me of the research of Energy-Based Model (EBM), where the objective is to learn the underlying unnormalized data distribution.\n\nA classical approach to EBM training is based on Maximum Likelihood Estimation, which involves alternating between updating the energy function and sampling from the given energy function. To accelerate the sampling process, Cooperative Training [1] introduces an auxiliary generator trained via MCMC-based teaching. Subsequent works [2, 3] extend this framework to multi-scale or multi-noise-level variants, which bear strong resemblance to the generator design adopted in this paper.\n\nImportantly, EBM training is typically conducted on highly complex, multimodal distributions—such as natural images or 3D shapes—where the energy functions are implemented as deep neural networks. In contrast, the distributions studied in this paper are relatively simple, fixed, and analytically tractable. Given appropriate hyperparameter settings, it is plausible that existing methods from [1, 2, 3] would already be capable of addressing the tasks considered in this paper.\n\nTherefore, I have concerns regarding the motivation and significance of the proposed approach.\n\n[1] Cooperative Training of Descriptor and Generator Networks\n\n[2] Learning energy-based models by cooperative diffusion recovery likelihood\n\n[3] Improving Adversarial Energy-Based Model via Diffusion Process\n\nQUESTIONS:\nNo extra questions\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSetting\n--\n\nThis is a method to train a continuous time Markov process in order to draw samples from the Boltzmann model:\n\np_{target} \\propto \\exp ( - E(x) )\nwith known energy E(x).\n\nThe goal is to have a single approach to generate  from continuous, discrete, mixed modalities. The method is demonstrated concretely to have as special cases diffusion, flow, and masked diffusion models. \n\nThe work builds on the recent paper of Holderrieth et al, ``Generator Matching'' [16], which\nalready unifies  diffusion models, flow matching, and discrete diffusion models;\nexplores jump processes, and allows for multimodal generation (continuous, discrete, mixed); but is data driven (vs the Boltzmann model in this submission).\n\nAs noted by [16], diffusion, flow, and jump processes are instances of continuous time Markov processes with a linear parametrization. The authors use F_t(x) to define the linear parametrization of a given infinitesimal generator which goes from noisy to clean.    The reparametrization trick (3) relates F_t to the conditional generator with parametrization F_{t|1}^{x_1}. The loss (4) gives a loss from the true F_t to the model F_t^{\\theta}. \n\nApproaches\n--\n\nA first approach is Energy-based Generator matching, where an approximation to F_t (x_t) is obtained with a self normalized importance weighted estimator with respect to distribution q_{1|t}, with the weights being a function of the unnormalized \\tilde p, and the normalization constant being computed as in (8). Eq. (4) then becomes eq. (9). \n\nThe challenge is in the design of q_{1|t}. A naive approach with high variance is to set it to p_{t|1}.  A second ``bootstrapping'' approch with lower variance is to use a nearby time r to t, under the weaker assumption  p_{r|t} \\approx p_{t|r}.  Eq. (14) gives the equation for L_t as a function of L_{t|r}^{x_r} and p_{r|t}.\n\nThen p_{r|t} is estimated using an importance sampler wrt q_{r|t}, which is chosen to be approximated by p_{t|r}, with lower variance.\n\nThis leads to another challenge: that the weights require \\tilde p_r (x_r). The estimator in eq. 18 has a similar high variance problem to the first approach, in that it approximates q_{1|r} \\propto p_{r|1}.  The model for \\tilde p_{r} is trained by noised energy matching.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n--\n\nthe idea is original (notwithstanding the dependence on [16], which is clearly and appropriately cited), the method is clearly explained, and performance on discrete and mixed continuous-discrete examples outperforms a baseline classical Gibbs sampler.\n\n\n\nMain theory limitation:\n--\n\nIt is not obvious to me that eq. 18 *provably* gets around the high variance problem, since it again makes the approximation  q_{1|r} \\propto p_{r|1} for a large jump in time from r to 1 (assuming t is close to r, to allow for small variance in that step).\n\nSo it seems you have just shifted the same problem to another part of the reasoning.\n\nIt is also not at all clear why, or if, the fact that the surrogate loss is used (eq. 20) to get a parametric model for \\tilde p_r solves the variance problem in a meaningful way since the loss is with respect to (19), which has high variance.  This would limit the accuracy with which one could accurately compute \\tilde p_r .\n\nCan you give a more formal arguemnt for why ``bootstrapping'' should be expected to guarantee lower variance? I understand that empirically it appears to do better, but I wonder if there is a theoretical argument to be made?\n\n\nMain experiment limitation:\n--\n\nThe evaluation was performed only on discrete (Ising) and joint discrete-continuous tasks, however the method also applies for the continuous setting. In order to evaluate the efficacy of the method, and in particular the issues that arise in eq. 18 and following, it is reasonable and necessary to benchmark the method against established methods (including those cited by the authors) for sampling from purely *continuous* EBMs (parallel tempering/SMC samplers/SOTA diffusion samplers).\n\nQUESTIONS:\nIs there a reason for eq. 5 being included? Is it used?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Energy-based Generator Matching (EGM), which extends the generator matching framework to scenarios where only energy functions are available rather than pre-computed equilibrium samples. The authors propose using self-normalized importance sampling to estimate the marginal generator in the absence of training data, along with a bootstrapping technique that leverages intermediate time steps with learned energy estimators to reduce variance in importance weights. EGM aims to provide a unified framework for training neural samplers across arbitrary continuous-time Markov processes (diffusion, flow, and jump processes) on general state spaces including continuous, discrete, and mixed modalities. The method is validated on several tasks including the 2D Ising model, Gaussian-Bernoulli RBM, and joint discrete-continuous sampling problems.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe paper tackles an important problem since many real-world applications naturally provide energy functions rather than equilibrium samples. I also appreciate the theoretical contribution of extending generator matching to energy-only settings, which addresses a practically important limitation. The unified framework supporting different Markov process types across general state spaces is conceptually valuable, and the mathematical development appears sound. The bootstrapping technique with Theorem 1 providing theoretical justification for using intermediate time steps is a clever contribution.\n\n# Weaknesses\nThat being said, I have serious concerns mostly about the experimental evaluation of this paper. \n\n- The authors of this paper fail to compare against key existing energy-based neural samplers. For example, with discrete tasks, LEAPS should have been compared, and for energy-based sampling, several relevant baselines exist.\n\n- Moreover, the experimental scope is insufficient for the claims made. The largest experiments are on 100-dimensional Ising models and 20-dimensional mixed problems, which doesn't demonstrate scalability to the high-dimensional problems where such methods would be most valuable. The authors acknowledge training instability issues but provide only ad-hoc solutions (exponential moving averages) without thorough analysis.\n- The technical quality is solid but not exceptional. The bias introduced by self-normalized importance sampling and potential proposal-posterior mismatch is mentioned but not rigorously analyzed. I think the paper would benefit from theoretical guarantees about convergence properties and bias bounds. The relationship between EGM and existing energy-based samplers like iDEM is unclear. For example, Equation (9) appears conceptually similar to existing importance sampling approaches in the energy-based sampling literature.\n\nQUESTIONS:\n- Missing Baseline Comparisons: Why weren't key energy-based neural samplers like iDEM, DDS, or PIS compared? These methods also work in energy-only settings and would provide much more meaningful baselines than Gibbs sampling alone. Can you provide experimental comparisons showing where and why EGM outperforms these existing approaches?\n\n- The paper acknowledges training instability and bias from self-normalized importance sampling but provides limited analysis. Can you provide theoretical or empirical analysis of: (a) convergence guarantees for the bi-level training scheme, (b) bias bounds for the SNIS estimator, and (c) conditions under which the method is expected to be stable?\n\n- Scalability Demonstration: The experiments are limited to moderate dimensions (≤100). Can you demonstrate the method's effectiveness on higher-dimensional problems where the advantages over traditional MCMC would be more compelling? What are the computational scaling properties compared to existing energy-based neural samplers?\n\n- Relationship to Existing Methods: When examining your estimator in Equation (9) and comparing to iDEM's approach, the techniques appear conceptually similar (both use importance sampling with energy-based weights). Can you clearly articulate the technical differences and when EGM would be preferred over existing methods?\n- While Theorem 1 justifies the bootstrapping theoretically, can you provide more rigorous analysis of when and why it reduces variance? Under what conditions might bootstrapping fail or introduce additional bias through the learned energy estimator?\n\nLIMITATIONS:\nNo limitations are discussed in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}