Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces
Authors: Tobias Schröder
Zijing Ou
Yingzhen Li
Andrew B. Duncan
Venue: NeurIPS 2024 poster (2024)
Keywords: Energy-based models
discrete probabilistic modelling
tabular data
Primary area: generative_models
TL;DR: A MCMC-free method based on the discrete heat diffusion for the training of energy-based models on discrete and mixed data with applications in generative modelling for tabular data.

Abstract: Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on data in discrete or mixed state spaces poses significant challenges due to the lack of robust and fast sampling methods. In this work, we propose to train discrete EBMs with Energy Discrepancy, a loss function which only requires the evaluation of the energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us to inform the choice of perturbation from the structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, we demonstrate the efficacy of the proposed approaches in a wide range of applications, including the estimation of discrete densities with non-binary vocabulary and binary image modelling. We also introduce the first application of EBMs to tabular data sets with applications in synthetic data generation and calibrated classification.

Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces
Tobias Schröder *t Imperial College London
Zijing Ou Imperial College London
Yingzhen Li Imperial College London
Andrew Duncan Imperial College London The Alan Turing Institute
Abstract
Energy-based models (EBMs) offer flexible framework for probabilistic mod- elling across various data domains. However, training EBMs on data discrete mixed state spaces poses significant challenges due to the lack of robust and fast sampling methods. In this work, we propose to train discrete EBMs with Energy Discrepancy, loss function which only requires the evaluation of the energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us inform the choice of perturbation from the structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, efficacy of the proposed approaches in a wide range of applications, including the estimation discrete densities with non-binary vocabulary and binary image mod- elling. Finally, we train EBMs on tabular data sets with applications in synthetic data generation and calibrated classification.
demonstrate the
Introduction
Discrete structures are intrinsic to most types of data such as text, graphs, and images. Estimating the data generating distribution Pdata of discrete data sets with a probabilistic model contribute greatly to downstream inference and generation tasks, and plays a key role in synthetic data generation of tabular, textual network data (Raghunathan, 2021). Energy-based models (EBMs) are probabilistic generative models of the form Pehm x (x)(- where the flexible choice of the energy function U allows great control in the modelling of different data structures. However, energy- models are, by definition, unnormalised models and notoriously difficult to train due to the intractability of their normalisation, especially in discrete spaces.
Energy-based models are typically trained with the contrastive divergence (CD) algorithm (Hinton, 2002) which performs approximate maximum likelihood estimation by approximating the gradient of the log-likelihood with Markov Chain Monte Carlo (MCMC) techniques. This method motivated rich research results sampling from discrete distributions enable fast and accurate estimation of energy-based models (Zanella, 2020; Grathwohl et al., 2021; Zhang et al., 2022b; Sun et al., 2022b,a, 2023a). However, the training of energy-based models with CD remains challenging relies
" Equul contributions Conrespondence to: t.schrodder22limpriaal... uk and z. ou2201merrial. uk CCoe: https://giis httpps:/ggthhbb... com/J
eneeeeetteeeeer---eeee maergyddiscrepenn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sufficiently fast mixing of Markov chains. Since accurate sampling from the EBM typically cannot be achieved, contrastive divergence lacks theoretical guarantees "Caareira-Perpinan & Hinton, 2005) and leads to biased estimates of the energy landscape (Nijkamp et al., 2019). For mixed data types, energy-based models have only been applied to combinations of numerical features with a single categorical label (Grathwohl et al., 2020).
The recently introduced Energy Discrepancy (ED) (Schröder et al., 2023) is new type of contrastive loss functional that, by definition, depends on neither gradients nor MCMC methods. Instead, the definition of ED only requires the evaluation of the energy function on positive and contrasting. negative samples which are generated by perturbing the data distribution. However, the work in Schröder et al. (2023) currently limited to Gaussian perturbations on continuous spaces and does not explores strategies to choose perturbations on discrete spaces, especially when these discrete spaces exhibit some additional structure.
In this work, we propose a framework to train energy-based models with energy discrepancy on discrete data, making the following contributions: 1) We explore a method to define discrete diffusion processes on structured discrete spaces through a heat equation on the underlying graph and investigate the effect of geometry and time parameter on the diffusion. 2) Based on the discrete diffusion process, we extend energy discrepancy to discrete spaces in a systematic way. thus introducing a MCMC- free method for the training of energy- based models that requires little tuning. 3) We extend our methodology to mixed state spaces and establish to the best of our knowledge the first robust training method of energy-baie mnase models on tabular data sets. We demonstrate promising performance on downstream tasks like synthetic data generation and calibrated prediction, thus unlocking a new tool for generative modelling on tabular data.
Preliminaries
Energy- baoegy-baseddio models (EBMs) are parametric family of distributions pe defined as
px(x)= exp(-Uo(x))
Zo = -0=Exe exp(-U
(1)
Zo
where Uo is the energy function parameterised by 0 and Zg denotes the normalisation constant. Ji=z J.W from unknown data distribution Pdata (x) we aim to learn an
Given set of Li.d. samples {x'
approximation (x) of Paata (x). The de facto standard approach for finding such is to minimise
the negative log-likelihood of po under the data distribution via gradient decent
V VelEpass(x)) [log pe (x) = Excepasta V VeL'o( (x)] - Ex_ ~po [VoUe(x )}.
(2)
The intuition behind this update is to decrease the energy of positive data samples x Priata (x) and to increase the energy of negative samples x ~ px)... However, the exact computation of gradient in (2) is known to be NP-hard in general (Jerrum & Sinclair, 1993) and quickly becomes prohibitive even on relatively simple data sets. Consequently, existing approaches resort to sampling from the model pe to approximate the gradient of bikeliiel via Monte Carlo estimation. In discrete settings, the most popular sampling methods include the locally informed sampler (Zanella, 2020), Gibbs with gradients (GwG) (Grathwohl et al., 2021), discrete Langevin (Zhang et al., 2022b), and generative flow networks (GFlowNet) (Zhang et al., 2022a). Despite their established success in discrete mergy-based modelling, these methods necessitate trade-off that hampers scalability: running the sampler for an extended duration rapidly increases the cost of maximum likelihood training, while shorter sampler runs yield inaccurate approximations of the likelihood gradient and introduce biases into the learned energy.
Energy Discrepancy (Schröder et al., 2023) is a recently proposed method to train energy-based -aased models without the need for an extensive sampling process. Instead, it constructs negative samples by perturbing the data, thus bypassing the sampling step while still yielding valid training objective. To elucidate, the energy discrepancy is formally defined as follows:
Definition (Energy Discrepancy). Let pasta be a positive density on a measure space (X, dx) and and let x be a conditional probability densiry. Define the contrastive potential induced by q as
U₃(y)= == -lo Σ (xtxx"xxp(-USS))
(3)
x'eX

---

The energy discrepancy between Pdata and U induced by q is defined as
(Paata: U) == E (xx (x) (xy))) (x
(4)
We will refer to q the perturbation. The validity of this loss functional was proven in Schröder et al. (2023) in large generality: In particular, it is sufficient for U = argmin ED (Paata, U) +> exp( - U² Pdata that any two points x, E X are q-equivalent, i.e. there exists chain of states (z))=1 E X with z = x, T y such that g(z (z') > O for all i , T-
Energy discrepancy can also be understood from seeing it as type of Kullback- Specifically, the loss function defined (4) is equivalent to the expected Kullback-Leibler divergence
divergence.
argmin (Dg((Patta U) <> argmin Exanca (y (KL(paataa- Pebm (-1x))
(S)
where P. (xly) x p-))((y)--) This relates energy discrepancy to diffusion recovery likelihood objectives (Gao et al., 2021) and Kullback-Leible contractions (Lyu, 2011). Energy discrepancy has demonstrated notable effectiveness in training EBMs in continuous spaces (Schröder et al., 2023). In the next section, we show how the loss can be defined in discrete spaces.
3 Energy Discrepancies for Discrete Data
For this work we will first consider a state space for the data distribution that can be written as the product of d discrete variables with S₁ classes each, i.e. X = =1,-11,...... Sk). Examples for spaces of this type are the categorical entries of data table for which dl denotes the number of features, or binary image data sets for which we typically write X' = {0, 1]d To define energy discrepancy on such spaces we need to specify a perturbation process under the following considerations: 1) The negative samples obtained through g are informative for training the EBM when only finite amounts of data are available. 2) The contrastive potential Ug has a numerically tractable approximation.
Let us consider one component X' f1..... 5), only. Inspired from previous works on diffusion modelling for discrete data (Campbell et al., 2022; Sun et al.. 2023b; Lou et al.. 2024; Campbell et al., 2024) we model the perturbation as a continuous time Markov chain (CTMC) with transition probability
(y b|r = a) = exp (t a, bE E (1,2,...,,S)
where R E Sx5 is the so-called rate matrix which satisfies Ros = - Eaze Roa and exp(tR) is the matrix exponential. For given rate matrix R, this approach then leaves us with a single tunable hyperparameter characterising the magnitude of perturbation applied. We first analyse how the choice of rate matrix and time parameter affect the statistical properties of the energy discrepancy loss. In fact, under weak conditions, the energy discrepancy loss converges to maximum likelihood estimation for - co, thus achieving the same loss function implemented by contrastive divergence:
Let gr (-(x)) be a Markov transition density defined by the rate matrix with eigenvalues 0 X1 (R) > X2(R) ... >s(R) and uniform stationary distribution. Then, there exists constant 24 independent of 0 such that mergy-discrepane converges to the maximum likeiihoooikeeiood loss
Theorem
DD (Peata: Uo) (0) zel S V/S
KL po).
with the loss of maximum- hikellhho-diee estimation (MLE(() -E (x) log px))]
Here, 2, is a constant independent of 0, so the optimisation landscapes of energy discrepancy estimation and maximum likelihood estimation in 0 align at an exponential rate, except for a shift by 24 which does not affect the optimisation. This result improves the lincar convergence rate in Schröder et al. (2023) and relates it to the spectral gap (22(R) of the rate matrix. Such result is meaningful as the maximum-likelln sample efficiency, and Theorem 1 suggests that energy discrepancy estimation approximate maximum likelihood estimation without resorting to MCMC like in classical EBM training methods. The proof is given in Appendix A.
estimator is generally statistically preferable with better
'On discrete spaces dx is assumed to be counting measure. On continuous spaces X, the appearing sums and expectations turn into integrals with respect to the Lebesgue measure 'With slight abuse of notations. we represent the contrastive potential induced by distribution as U, and denote the energy function U with without the subscript 0.
3

---

Ra
Figure 1: Visualisation of a typical state space of tabular dataset: Numerical entries taking values in R cyclical categorical entries (e.g. season), ordinal categorical entries (e.g. age), unstructured categorical entries, and variables with absorbing state associated with masking the entry.
3.1 Heat Equation in Structured Discrete Spaces
In principle, Theorem establishes that energy discrepancy converges to the loss of maximum likelihood estimation the limit + oc for any choice of rate matrix with spectral gap. In practice, however, large perturbations of data can produce high- Sigh-variance parameter gradients and provide little training signal. Instead, it is desirable construct perturbations that allow a fine-grained trade-off' between the statistical properties of the loss function and the variance of the gradients. For this reason, we investigate the perturbation for small which, as we will see, can be informed by the assumed graph structure of the underlying discrete space.
The infinitesimal perturbations of the CTMC characterised by the rate matrix. Notice that for small I, the Euler discretisation of the heat equation yields
ge (y b/æ a) = 5(b, a) + t.
with 5(b, =
b and zero otherwise. To model the relationship between values b E {I....,S) .} we endow the space with a graph structure with adjacency matrix A and out degree matrix Dout and model the rate matrix the graph Laplacian R == (A Dout). ). By definition, the columns of the graph Laplacian matrix sum to zero Σ5 Rix The smallest possible perturbation is then characterised as the transition to an adjacent neighbour. The characterisation of the CTMC in terms of the graph Laplacian is implicitly assumed in previous work. Campbell al. (2022) describe diffusion via uniform perturbation which corresponds to a fully connected graph and Lou et al. (2024) describe the rate matrix associated to a star graph with absorbing (masking) state D:
Runif 1 S 5bb a):
Rmmask
5(b, 5(b, a).
For visualisation, see Section 3.1. In addition to these fairly unstructured rate matrices model state spaces with cyclical or ordinal structure:
2
0
2
0
neve
- 1
0
Rord
-2 1
0
We restrict ourselves to uniform, cyclical, and ordinal structures as these structures can typically be trivially inferred from the type of data modelled. For example, periodically changing quantities (c.g. seasons) would display cyclical structure and ordered information like age an ordinal structure. It is possible, however, extend our framework arbitrary graphical structures of the state space as long as eigenvalue decompositions of the graph Laplacian are feasible.
Since the Gaussian perturbation on Euclidean space used in Schröder et al. (2023) is also the solution of a heat equation, these choices allow us to model the perturbation on a vector of mixed entries including numerical values, unstructured categorical values, and structured categorical values with single differential equation
0.ge(- (-Ix) (A sum & RE Hord
Runif
Rabbs) qc(-1x),
go(-1x)
6(-,x)
(6)
where A num denotes the standard Laplace operator Σd Ei= 02 / and the product ( denotes that
each operator acts on the corresponding component of the state space.

---

Estimating the Energy Discrepancy Loss
We now discuss how discrete energy discrepancy can be estimated. We will typically assume that each dimension of the data point is perturbed independently, i.c. the perturbation g(y\x) is modelled perturbations. On Euclidean data, we resort to the implementation in Schröder et al. (2023) and obtain perturbed samples by adding isotropic Gaussian noise to the samples. We are now left with the heat equation on discrete space.
as the product of component-we
4.1 Solving the Heat Equation
In the case of the uniform Laplacian Runit = 15 - Sid, with 1y denoting an all ones vector of length S, the heat equation has the closed form solution
1 e St
q. (yll a) e (555 a)
Σ 8(y.
S
Practically speaking, this perturbation remains in its state with probability e St from the state space otherwise. The case of the cyclical and ordinal structure is more delicate. We first note that the heat equation can be solved in terms of its eigenvalue expansion exp(Rt) = V e'ex(AA) where A is the matrix with the eigenvalues Ap along its diagonal and V is matrix of orthogonal eigenvectors with each column containing the corresponding eigenvector vg. The perturbation for H and Hord can then be computed by means of discrete Fourier transform:
St and samples uniformly
Proposition Assume the density qe (bla)) == (e b|r a) is defined by the rate matrices R or Rord The transition density for all o, b (1,2...., {} is given by
1
(x0(bla)
Σ
exp(2ribuu)") exp ((2 (2TW 2)() exp( 1pp(-2nt -2riiou (
(8)
p=1
go (bla)
2
Σ
1
cos((26: - 1)wpprr ord) exp (20((22 - -2)( cos((2u- 1) 1)TW
(9)
5
2p
p=1
where wpre = (p - S and word = =(p 1)/25, respectively, and xp = 12.,,...,,)
For the derivation, see Appendix A.2. Due to this result, the heat equation can be efficiently solved in parallel without requiring any sequential operations like multiple Euler steps. In addition, the transition matrices can be computed and saved in advance, thus reducing the computational complexity to the matrix multiplication with a batch of one-hot encoded data points.
Gaussian limit and choice of time parameter. For tabular data sets the cardinality S changes between different dimensions which raises the question how t should be scaled with S. To answer this question observe the following scaling limit of the perturbation:
Theorem 2 (Scaling limit). Let Ya qe(-lx μS) with μ. E (1/S, 2/SSS ...., 1}, where is either the transition density of the cyclical or ordinal perturbation. Let 4 R (0, 1), where for all E Z
and x E (0, 1] peve (n x) = x and gord (2n + x) I, aard (2n ++++) -x. Then,
S S-00 (pE,) with & N~,, 2t).
Consequently, under the rescaling of time and space prescribed, the perturbation behaves indepen- dently of the state space size like Gaussian with variance 2t that is reflected or periodically continued at the boundary states (0, 1]. The phenomenon is visualised in Figure 5. Based on this scaling limit we typically choose quadratic rule f = S2t Alternatively,
may choose linear rule t t= Staase in which case the limit becomes a regular Gaussian on R,, thus recovering the Euclidean case from Schröder al. (2023). The theorem is proven in Appendix A.3.
As a byproduct of this result we can also approximate the perturbation with discretised rescaled samples from standard normal distribution and applying either periodic or reflecting mappings on perturbed states outside the domain. This may be computationally favourable for spaces of the form {l...., 5)d where the vocabulary size S and dimension of the state space grow very large.
5

---

Localisation to random grid. For unstructured categorical variables the uniform perturbation may introduce too much noise to inform the EBM about the correlations in the data set. In these cases, can be beneficial to sample random dimension k E (1,..., d) and apply a larger perturbation in this dimension, only. This effectively means to replace the product of categorical distributions with a mixture perturbation
1 Σ qe (Ukc
2: (y(xx)
II qe (v< - A=1
k=1
In our experiments we only consider the case of perturbing the randomly chosen dimension uniformly. We call this grid perturbation due to connections with concrete score matching (Meng et al., 2022). The resulting loss can be understood
variation of sneudo-likelihodd estimation.
Special case of binary state space. In the special case of X {0, 13dd, the structures of the cyclical, ordinal, and uniform graph coincide, and the perturbation q: (y x) becomes the product of identical Bernoulli distributions with parameter ε 0.5(1 which assumes that dimension is selected at random and the entry is flipped deterministically from zero to one or one to zero. For details, see Appendix B.1.
2t), ). We also explore the grid perturbation
4.2 Estimation of the Contrastive Potential
The final challenge in turning energy discrepancy into a practical loss function lies in the estimation of the contrastive potential Ug. We use the fact that for symmetric rate matrix R, the induced perturbation is symmetric as well, i.c. (e(y/x) = qe (xly). Thus, we first write the contrastive poten- tial as an expectation Uy(y) = log Exex exp( ()p((((())e) (x)))q(())x = llog Eg(x y) and subsequently approximate the energy discrepancy loss as in Schröder et al. (2023) as N (w.,,u(()) = N E log (w +
Σラ and X3x ~ (e The details of the training procedure are provided in Appendix C.
M exp(U((x) - (x-*)) with x² Pdttaa y (-(-)x))
Related Work
Contrastive loss functions. Our work is based on energy discrepancies first introduced in (Schröder al., 2023). Energy discrepancies are equivalent to certain types of KL contraction divergences whose theory was studied in Lyu (2011), however, without proposing training algorithm for EBM's. s. On Euclidean data, ED related to diffusion recovery-liked likelihood (Gao et al., 2021) which uses CD-type training algorithm. For a masking perturbation, ED estimation can be understood as Monte-Carlo approximation of pseudo-likelihood (Besag, 1975). Furthermore, the structure of the stabilised energy discrepancy loss shares similarities with other contrastive losses such as Ceylan & Gutmann (2018); Gutmann Hyvärinen (2010); Oord et al. (2018); Foster et al. (2020) due to their close connection to the Kullback-Leible divergence.
Discrete diffusion models. We extend the continuous time Markov chain framework introduced and developed in Campbell et al. (2022, 2024); Lou et al. (2024) and provides a geometric interpretation thereof. Similar to us, Kotelnikov et al. (2023) defines a flow on mixed state spaces as the product of Gaussian and categorical flow, utilising multinomial flows (Hoogeboom et al., 2021). Our work has connections to concrete score matching (Meng et al., 2022) through the usage of neighbourhood structures to define replacement of the continuous score function.
Contrastive divergence and sampling. Contrastive divergence (CD) is commonly utilised for training energy-based models in continuous spaces with Langevin dynamics (Xie et al., 2016, 2018, 2022; Du et al., 2021; Xiao et al., 2020). In discrete spaces, EBM training heavily relies on CD methods as well, which is a major driver for the development of discrete sampling strategies. The standard Gibbs method was improved by Zanella (2020) through locally informed proposals. This method was extended to include gradient information (Grathwohl et al., 2021) to drastically reduce the computational complexity of flipping bits in several places (Sun et al., 2022b; Emami et al., 2023; Sun et al., 2022a). Moreover, a discrete version of Langevin sampling was introduced based on this idea (Zhang et al., 2022b; Rhodes & Gutmann, 2022; Sun et al., 2023a). Consequently, most current implementations of contrastive divergence use multiple steps of a gradient-based discrete sampler. Alternatively, EBMs can be trained using generative flow networks which learn Markov chain that construct data optimising the energy reward function (Zhang et al., 2022a).

---

Figure 2: Comparison of energy discrepancy and contrastive divergence on the dataset with 16 dimen- sions and 5 states. Rows and show the estimated density and synthesised samples, respectively.
Other training methods and applications of EBMs for discrete and mixed data. A sampling- free approach for training binary discrete EBMs is ratio matching (Hyvärinen, 2007; Lyu, 2009; Liu et al., 2023). Dai al. (2020) propose apply variational approaches train discrete EBMs instead of MCMC. Eikema et al. (2022) replace the widely-used Gibbs algorithms with quasi-rejection sampling trade off the efficiency and accuracy of the sampling procedure. The perturb-and-map (Papandreou & Yuille, 2011 I) is also recently utilised to sample and learn in discrete EBMs LLazaro-Gredill et al., 2021). Tran et al. (201 1) introduce mixed-varies restricted Boltzmann machines for energy-based modelling on mixed state spaces. Deep architectures, on the other hand, have been mostly limited to single categorical target variable which modelled via classifier (Grathwohl al., 2020). Moreover, Ou et al. (2022) apply discrete EBMs on set function learning, in which the discrete energy function is learned with approximate marginal inference (Domke, 2013).
Experiments
To evaluate our proposed approach, we conduct experiments across diverse scenarios: i) estimating probability distributions discrete data; ii) handling mixed-state features in tabular data; and iii) modelling binary images. We also explore Ising model training and graph generation in binary spaces, but leave the detailed evaluation of these in Appendix D.
6.1 Discrete Density Estimation
We first demonstrate the effectiveness of energy discrepancy on density estimation using synthetic discrete data. Following Dai et al. (2020), we initially generate 2D floating-point data from several two-dimensional distributions. Each dimension of the data is then converted into a 16-bit Gray code, resulting dataset with 32 dimensions and states. To construct datasets beyond binary cases, we follow Zhang et al. (2024) and transform each dimension into 8-bit 5-base code and 6-bit decimal code. This process creates two additional datasets: one with 16 dimensions and 5 states, and another with 12 dimensions and 10 states. The experimental details are given in Appendix D.I.
Figure 2 illustrates the estimated energies well as samples that are synthesised with Gibbs sampling for energy discrepancy (ED) and contrastive divergence (CD) on the dataset with 16 dimensions and 5 states. It can be seen that ED excels at capturing the multi-modal nature of the distribution, consistently learning sharper energy landscape in the data support compared to CD. This coincides with the previous observations in continuous spaces (Schröder et al., 2023), suggesting ED's advantage in handling complex data structures. For more results of additional datasets with 5 and 10 states, we deferred them to Figures 7 and 8, respectively.
For binary cases with 2 states, we compare our approaches to three baselines: PCD (Tieleman, 2008), ALOE+ (Dai al., 2020), and EB-GFN (Zhang et al., 2022a). In Tables 3 and 4, we quantitatively evaluate different methods by evaluating the negative likeliied (NLL) and the exponential Hamming MMD (Gretton et al., 2012), respectively. We observe that energy discrepancy outperforms the baseline methods in most settings, but without relying MCMC simulations (as in PCD) or the training of additional variational networks (as in ALOE and EB-GFN). This performance gain is likely explained by the good theoretical guarantees of energy discrepancy for well-posed estimation tasks. In contrast, the baselines introduce biases due to their reliance on variational proposals and short-run MCMC sampling that may not have converged.
7

---

Table 1: Results on real- datasets.
Adult
Bank
Cardio
Churn
Mushroom
Beijing
Avg. Rank
Methods
AUC
AUC
AUC
1
AUC
RMSE 4
Real
227
.935.0002
.81
.819
1.00
.23+.003
CTGAN
861+.00
7706
.787..002
.792..00 0033
7877
1.01
6.33
TVAE
.8682.002
.676=.009
.793=.006
.9992.000 000
1.05#.012
5.17
TabCD
.6122.02
.6042.02 .021
.7652.008
5584=.22 021
.5612.04
1. 1.062.037
8.83
TahDDPM
99002000 0011
.922= 001
.81 +001
.806..00 007
.999 000
.5562.005
1.5
TabED-Uni
8844..om
.842 .013
.786+.002
.810=.0xx
.998+.001
1.04+.013
3.83
TabED-Grid
.833+.00
.801+.004
.791+.001
.803=.007
.985..005
+978+.015
4.83
TabED Cyc
.823=.007
.7962.001
.807
.971 004
1.01
4.83
TabED- Ord
853
.8452.001
.7922.002
.8062.00 004
210
1.02
4.83
TabED-Str
.899.004
004
.8192.001
.978+.012
3.67
6.2 Tabular Data Synthesising
In this experiment, assess our methods synthesising tabular data, which presents challenge due to its mix of numerical and categorical features, making it more difficult to model compared to conventional data formats. To demonstrate the efficacy of energy discrepancies, first conduct experiments on synthetic examples before proceeding to real-world tabular data. Additional details regarding the experimental setup are deferred to Appendix D.2.
Synthetic Dataset. We first showcase the effectiveness of our methods on mixed data types by learning EBMs on synthetic ring dataset. The dataset consists of four columns, with the first two columns indicating numerical coordinates of data points. The third column categorizes data points into four circles whereas the last column speci- fies the 16 colours each data point could be classified into. Therefore, each row in the tabular contains 2 numerical features and categorical features.
Data
CD
ED
Figure 3: Comparison of the energy dis- crepancy and contrastive divergence on the synthetic tabular datasets.
To train an EBM on dataset comprising mixed types of data, we employ either contrastive divergence energy discrepancy. For CD, we adopt strategy involving a replay buffer in conjunction with short-run MCMC using 20 steps. Specifically, we utilise Langevin dynamics and Gibbs sampling for numerical and categorical features, respectively. In the case of ED, we perturb the numerical features with a Gaussian perturbation and the categorical features with grid perturbation. Figure illustrates the results of synthesised samples generated from the learned energy using Gibbs sampling. These findings align with those depicted in Figure 2, where CD struggles to capture a faithful energy landscape, leading to synthesized samples potentially lying outside the data distribution support. Instead, by leveraging a combination of perturbation techniques tailored to the data types present, ED offers more robust and reliable framework for training EBMs mixed state spaces.
Real-world Dataset. We then evaluate our methods by benchmarking them against various baselines across 6 real-world datasets. Following Xu et al. (2019), we first split the real datasets into training and testing sets. The generative models are then Iearned on the real training set, from which synthetic samples of equal size generated. This synthetic dataset is subsequently used to train classifccatinnregression XGBoost model, which evaluated using the real test set.
We compare the performance, as measured by the AUC score for classification tasks and RMSE for regression tasks, against CTGAN, TVAE, (Xu et al., 2019) and MabDDPM (Kotelnikov al., 2023) baselines which utilise generative adversarial networks, variational autoencoders, and denoising diffusion probabilistic models, respectively. The results are reported Table 1. Here, TabED-Str refers to an ED loss for which the perturbation was chosen with prior knowledge about the structure of the modelled feature, i.e. ordinal and eyclical features were hand-picked. We not report results for TabED-Str on the Cardio, Churn, and Mushroom datasets, since the state spaces only consist

---

Table 2: Experimental results for discrete image modelling. We report the negative log- log-likelihod (NLL) on the test set for different models. The results of Gibbs, GWG, and DULA are taken from Zhang et al. (2022b), and the result of EB-GFN from Zhang et al. (2022a).
Dataset \\n
Method
Gibbs
GWG
EB-GFN
DULA
ED-Bern
ED-Grid
Static MNIST
117.17
80.01
102.43
80.71
96.11
90.61
Dynamic MNIST
121.19
80.51
105.75
81.29
97.12
90.19
Omniglot
142.06
94.72
112.59
145.68
97.57
93.94
of unstructured features. To compute the average ranking datasets since on unstructured features TabED-Uni and TabED-Str coincide.
the rank of TabED-Uni on these
The variants of ED show promising results on diverse datasets, thus demonstrating the suitability of ED for EBM training on mixed-state spaces. While TDDDDPM outperforms the other approaches, TabED shows comparable performance to the CTGAN and TVAE baselines and outperforms both in average ranking. Furthermore, the contrastive divergence approach performs poorly which highlights its limitations in accurately modelling distributions on mixed state spaces. Surprisingly, the unstructured perturbation TabED-Uni performs slightly better than the structured approaches. This may partially be attributed to the fact that the state spaces of the discrete features are relatively small. Consequently, the uniform perturbation might be a good approximation of maximum likelihood estimation in agreement with Theorem I, while not producing high-variance gradients on these specific datasets.
Improving Calibration. Despite the improving accuracy of neural- network-based classifiers in recent years, they are also becoming increas- ingly recognised for their tendency to exhibit poor calibration due to over- confident outputs (Guo et al., 2017; Mukhoti et al., 2020). Since energy- based model on mixed state spaces can capture the likelihood of tuples of features and target labels, they im- plicitely quantify the confidence in a prediction and can be adapted into classifiers with better calibration than deterministic methods. This opens up new avenue for applying EBMs in deterministic tabular data modelling methods.
uxpectd ure Jota
cant
Cor Mdrroe
Figure 4: Calibration results comparison between the base- line (left) and energy discrepancy (right) on the adult dataset.
Let and x be the target label and the rest features in the tabular data, an EBM Ug v) learned
on the joint probability Pdata (), can be transformed into deterministic classifier: pEBM (y)x) o exp(-Ue(x, (x. 3)). As baseline for comparison, we additionally train classifier Pc x) with the same architecture by maximising the conditional likelihood: Epexe, (logP (c Results on the adult dataset can be seen in Figure 4. We find that the EBM and the baseline exhibit comparable accuracy. However, the baseline model is less calibrated, generating over-confident predictions. In contrast, the EBM learned through ED achieves better calibration, as evidenced by lower expected calibration error (Guo et al., 2017). Further details and results are provided in Appendix D.2.
6.3 Discrete Image Modelling
In this experiment, we evaluate our methods on high-dimensional binary spaces. Following the settings in Grathwohl et al. (2021), we conduct experiments on various image datasets and compare against contrastive divergence using various sampling methods, namely vanilla Gibbs sampling, Gibbs-Wing Bibbs-With-GGr With-Gradient (Grathwohl et al., 2021, GWG), Seoeraaive-loo-Neetwwe GFN), and Discrete-Unann Laanjjusttd-Laaeviinnn details are provided in Appendix D.3. After training, annealed importance sampling (Neal, 2001) is employed to estimate the negative log- ligeliieli (NLL).
(Zhang et al., 2022a, (Zhang et al., 2022b, DULA). The training
Table 2 displays the NLLs on the test dataset. parable performance to the baseline methods on the Omniglot dataset. Despite the performance gap compared to the contrastive divergence methods on the MNIST dataset, energy discrepancy stands out for its efficiency, requiring only M evaluations of the energy function in parallel (see
evident that energy discrepancy achieves com-

---

Table 7 for the comparison of running time complexity). This represents significant computational reduction compared contrastive divergence, which lacks the advantage parallelisation and involves simulating multiple MCMC steps. Additionally, our methods show superiority over CD-1 by a substantial margin, as demonstrated in Table affirming the effectiveness of our approach. For further insights, we provide visualisations of the generated samples in Figure 10.
7 Conclusion and Limitations
In this paper we extend the training of energy-based models with energy discrepancy to discrete and mixed state spaces in systematic way. We show that the energy-basee model can be learned jointly on continuous and discrete variables and how prior assumptions about the geometry of the underlying discrete space be utilised in the construction of the loss. Our method achieves promising results on a wide range of discrete modelling applications at a significantly lower computational cost than MCMC- approaches. To the best of our knowledge, our approach is also the first working training method for energy-based models on tabular data sets, unlocking a wide range of inference applications for tabular data sets beyond the scope of classical joint energy-basee models.
Similar to prior work on energy discrepancy in continuous spaces (Schröder et al., 2023), our training method is sensitive to the assumption that the data distribution is positive the whole state space. While our method scales to high-dimensional datasets like binary image data, where the positiveness of the data distribution is assumed to be violated due the manifold hypothesis, the large difference between intrinsic and ambient dimensionality poses challenges to our approach and may explain why energy discrepancy cannot match the performance of contrastive divergence with large number of MCMC steps on binary image data.
Limitations:
Broader Impact: In principle, our method can be used for imputation and prediction in tabular data sets and can thus have discriminating or excluding effects used irresponsibly.
Outlook: For future work, we are interested in extensions to highly structured types of data such as molecules, text, or data arising from networks. So far, our work only considers cyclical and ordinal structures on the discrete space, while incorporating more complex structures as prior information into the rate function may be beneficial. Furthermore, interesting downstream applications ranging from table imputation with confidence bounds, simulation-bast inference involving discrete variables, or reweighting of language models with residual EBMs have been left unexplored this work.
Author Contributions
TS and ZO conceived the project idea to use an ED loss for the training of EBMs on discrete and mixed data. TS devised the main conceptual ideas, developed the theory, conducted the proofs and implemented the ED loss. ZO contributed to the conceptual ideas and designed and carried out the experiments. ABD supervised the conceptualisation and execution of the research project and contributed proof ideas. ZO, YL, and ABD checked derivations and proofs. TS and ZO equally contributed to the writing under the supervision of YL and ABD.
Acknowledgements
TS was supported by the EPSRC-DTP scholarship partially funded by the Department of Mathematics, Imperial College London. ZO was supported by the Lee Family Scholarship. We thank the anonymous reviewer for their comments.
10

---

References
Anderson, W. J. Confinuous-time Markov chains: An applications-oriented approach. Springer Science & Business Media, 2012.
Besag, J. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society. Series (The Statistician), 24(3):179-199) 1975.
Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-2822 28279, 2022.
Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows discrete spaces: Enabling multimodal flows with applications to protein co-design. International Conference Machine Learning, 41, 2024.
Perrerra-Perpinant M. and Hinton, G. On contrastive divergence learning. International workshop on artificial intelligence and statistics, pp. 33-40. PMLR, 2005.
Ceylan, C. and Gutmann, M. U. Conditional contracontrastive estimation of unnormalised models. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018.
Dai, H., Singh, R., Dai, B., Sutton, C., and Schuurmans, D. Learning discrete energy based models via auxiliary-variable local exploration. Advances in Neural Information Processing Systems, 33: 10443-10455, 2020.
Diaconis, P. and Stroock, D. Geometric bounds for eigenvalues of markov chains. The annals of applied probability, pp. 36-61, 1991.
Domke, J. Learning graphical model parameters with approximate marginal inference. IEEE
transactions on pattern analysis and machine intelligence, 35(10))2454-245) 2013.
Du, Y., Li, S., Tenenbaum, J., and Mordatch, I. Improved contrastive divergence training of energy based models. In International Conference on Machine Learning, 2021.
Eikema, B., Kruszewski, G., Dance, C. R., Elsahar, H., and Dymetman, M. An approximate sampler for energy-basee models with divergence diagnostics. Transactions of Machine Learning Research, 2022.
Emami, P., Perreault, A., Law, J., Biagioni, D., and John, P. Plug play directed evolution of proteins with gradient-based discrete MCMC. Machine Learning: Science and Technology, 4(2), 2023.
Foster, A., Jankowiak. M.. O' O'Mear,, M., Teh, Y. W., and Rainforth, T. A unified stochastic gradient approach designing bayesian-optimal experiments. In International Conference on Artificial Intelligence and Statistics, pp. 2959-29999 2969. PMLR, 2020.
Gao, R., Song. Y., Poole, B., Wu, N., and Kingma, D. P. Learning energy models by difffusion recovery likelihood. International Conference on Learning Representations, 2021.
Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K. Your classifier is secretly an energy based model and you should treat like one. International Conference on Learning Representations, 2020.
Grathwohl, W., Swersky, K., Hashemi, M., Duvenaud, D., and Maddison, C. J. Oops I took a gradient: Scalable sampling for discrete distributions. In International Conference Machine Learning, volume 38, 2021.
Gray, F. Pulse code communication. United States Patent Number 2632058, 1953.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, kernel two-sample test. The Journal of Machine Learning Research, 13(1), 2012.

---

Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Interaational conference on machine learning, pp. 1321-1330. PMLR, 2017.
Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirleenth international conference artificial intelligence and statistics, pp. 297-304, JMLR Workshop and Conference Proceedings, 2010.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Hinton, E. Training products experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Hoogeboom, E.,, Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465, 2021.
Hyvärinen, A. Some extensions of score matching. Computational statistics & data analysis, 51(5): 2499-2512, 2007.
Jerrum, M. and Sinclair, A. Polynomial-imme approximation algorithms for the ising model. SIAM Journal on computing, 2275)1087 1993.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Kipf, T. N. and Welling, M. Semi supervipd classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 2023.
PMLR,
Lazaro-Gredilla, M., Dedieu, A., and George, D. certur--andmmax-rrodue Sampling and learning in discrete energy-based models. Advances in Neural Information Processing Systems, 34:928-940, 2021.
Li, Y., Vinyals, O., Dyer, C., Pascanu, R., and Battaglia, P. Learning deep generative models of graphs. In International Conference Machine Learning, 2018.
Liu, J., Kumar, A., Ba, J., Kiros, J., and Swersky, K. Graph normalizing flows. Advances in Neural Information Processing Systems, 32, 2019.
Liu, M., Liu, H., and Ji, S. Gradient-guided importance sampling for learning binary energy-based based models. In International Conference on Learning Representations, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
Losonezi, L.. Figenvalues and eigenvectors of some tridiagonal matrices. Acto Mathematica Hungar- ica, 6003):30--32,, 1992.
Lou, A., Meng. C., and Ermon, S. Discrete diffusion language modeling by estimating the ratios of the data distribution. International Conference on Machine Learning. 41, 2024.
Luo, Y., Yan, K., and Ji, S. Graphdf: A discrete flow model for molecular graph generation. In International Conference on Machine Learning, pp. 7192-7203, PMLR, 2021.
Lyu, S. Interpretation and generalization of score matching. In Uncertainty Artificial Intelligence, volume 25, 2009.
Lyu, S. Unifying non-maximum likelihood learning objectives with minimum kl contraction. In Shawe- Taylor, J., Zemel, R., Bartlet, P., Pereira, F, and Weinberger, (eds.), Advances Neural Information Processing Systems, volume 24, Curran Associates, Inc., 2011.
12

---

Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, volume 35, 2022.
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., and Dokania, P. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33: 33:15288-15299 2020.
Neal, R. M. Annealed importance sampling. Statistics and computing, 11, 2001.
Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. Learning non- non-convergent non non-persistent short-run MCMC toward energy- based model. In Neural Information Processing Systems, volume 33, 2019.
Niu, C., Song, Y., Song. J., Zhao, S., Grover, A., and Ermon, S. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, pp. 4474 4484. PMLR, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arviv:
2018
Ou, Z., Xu, T., Su, Q Li, Y., Zhao, P., and Bian, Y. Learning neural set functions under the optimal subset oracle. Advances in Neural Information Processing Systems, 35:35021-35033 2022.
Papandreou, G. and Yuille, A. L. Perturb-and-map random fields: Using discrete optimization to learn and sample from energy models. In 2011 International Conference on Computer Vision, pp. 193-200. IEEE, 2011.
Raghunathan, T. Synthetic data. Annual review of statistics and its application, 8:129-144 140, 2021.
Raginsky, M. Strong data processing inequalities and e-sobolev inequalities for discrete channels. IEEE Transactions on Information Theory, 62(6):3355 -313335553399,,, 2016.
Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv: 17X1vv1710.05944 2017.
Rhodes, B. and Gutmann, M. U. Enhanced gradient- madde meme in discrete spaces. Transactions on Machine Learning Research, 2022.
Schröder, T., Ou, Z., Lim, J., Li, Y., Vollmer, S., and Duncan, A. Energy discrepancies: score- independent loss for energy-based models. Advances Neural Information Processing Systems, 36, 2023.
Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T. Collective classification in network data. Al magazine, 29(3):93-93, 2008
Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M., and Tang, J. Graphaf: a flow-based autoregressive model for molecular graph generation. International Conference Learning Representations, 2020.
Simonovsky, M. and Komodakis, N. Graphvae: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Learning-ICANT 2018: 27th Interna- tional Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part 27. Springer, 2018
Sun, H., Dai, H.. and Schuurmans, D. Optimal scaling for locally balanced proposals in discrete spaces. In Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022a.
Sun, H., Dai, H., Xia, W., and Ramamurthy, A. Path auxiliary proposal for MCMC in discrete space. In International Conference on Learning Representations, 2022b.
Sun, H., Dai, H., Dai, B., Zhou, H., and Schuurmans, D. Discrete Langevin samplers via Wasserstein gradient flow. In International Conference on Artificial Intelligence and Statistics. PMLR, 2023a.
Sun, H., Yu, I.., Dai, B., Schuurmans, D., and Dai, Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representationss 2023b.
13

---

Tee, G. J. Eigenvectors of block circulant and alternating circulant matrices. New Zealand Journal of Mathematics, 36(8):195-211, 2007.
Tieleman, Т. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference Machine learning, 2008
Tran, T., Phung. D., and Venkatesh, S. Mixed-varian variate restricted boltzmann machines. In Asian conference on machine learning, pp- 213-229. PMLR, 2011.
Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A. Vaebm: symbiosis between variational autoencoders and energy-based models. International Conference on Learning Representations, 2020.
Xie, J., Lu. Y., Zhu, S.-C., and Wu, Y. theory of generative convnet. International Conference on Machine Learning. pp. 2635-2644, PMLR, 2016.
Xie, J., Lu, Y., Gao, R., and Wu, Y. N. Cooperative learning of energy barggybbsedd model and latent variable model via MCMC teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Xie, J., Zhu, Y., Li, J., and Li, P. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model. In International Conference on Learning Representations, 2022.
Xu, L., Skoularidou, M., Cuesta-Infante A., and Veeramachaneni, K. Modeling tabular data using conditional gan. Advances in neural information processing systems, 32, 2019.
You, J., Ying, R., Ren, X., Hamilton, W., and Leskovec, J. Graphrnn: Generating realistic graphs with deep augo-regressive models. In International conference on machine learning, pp. 5708-5717. PMLR, 2018.
Yueh, W.-C. Eigenvalues of several tridiagonal matrices. Applied Mathematics E-Notes (electronic only]. 5:66-74, 2005.
Zanella, G. Informed proposals for local MCMC in discrete spaces. Journal of the American Statistical Association, 1 115(530)8552866, 2020.
Zhang, D., Malkin, N., Liu, Z., Volokhova, A., Courville, A., and Bengio, Y. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning. pp. 26412-2600 26428. PMLR, 2022a.
Zhang, P., Yin, H., Li, C., and Xie, X. Formulating discrete probability flow through optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, R., Liu, X., and Liu, Q. A LLanevvin-like sampler for discrete distributions. In International Conference on Machine Learning, pp. 26375-26396. PMLR, 2022b.
14

---

Appendix for "Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces"
Contents
Proofs of the Main Results
15
A.1 Proof of Theorem
15
A.2 Eigenvalue Decomposition of Rate Matrices for Proposition
16
A.3 Proof of Scaling limit Theorem 2
17
B Estimation of Energy Discrepancy
18
B.1 Binary Case
20
B.2 Connection pseudo-likelinge likelihood estimation
20
Tabular Data Synthesising with Energy-Based Models
21
D Additional Experimental Results
23
D.1 Discrete Density Estimation
23
D.2 Tabular Data Synthesising
24
D.3 Discrete Image Modelling
................................
25
D.4 Training Ising Models
27
D.5 Graph Generation
27
E Naming Conventions and Parameters of Introduced Methods
29
A Proofs of the Main Results
A.1 Proof of Theorem 1
Theorem Let qe(-(z) be a Markov transition density defined by the rate matrix with eigenvalues
0 = (R) 2 A2(R) 2 ... 2 Ag(R) and uniform stationary distribution. Then, there exists
constant 2 independent of such that energy-discrepancy converges to the maximum likelihood moximum-likelhhood loss
FEDg qt (Pdata: Uo) (MLL(() =1 S exp((1222(P)) KT Po).
with the loss of cinsiimmm-lkkelhhone estimation (MLE(O) == - (x) log (x)).
Proof. For X' = -1,2,.... and two probability distributions yr and p2 on define the marginal distributions
QuP((v) = Σ (1)) QiPZ(u) Σ (((l)))p(r)
(10)
ZEX
ZEX
By the data-processing inequality holds for all pr, p2 with KL(Pi P2) < oo that
(T(QQ.P || Q.Pz) S (T))KT(P)) P2) with <(l) $
(11)
We are going to bound the contraction rate c(t) by firstly making use strong data processing inequality Raginsky (2016) which states that the contraction rate can be bounded by the Dobrushin contraction coefficient
<(t) ((Q,) supllQ Q+6", IV
(12)
15

---

Furthermore, since total variation is metric, the contraction between two points z, 2' is upper bounded by the contraction towards the stationary distribution the CTMC
Qupl - Q:O'IITV S 22supl Qró, - T||TV
(13)
Next, use Diaconis & Stroock (1991, Proposition 3) which states that for any E X and for eigenvalues 0 = (R) 2 (R( 2 ... 2 (R)
11Qu0x -lfv - 11 (x) 4 T(z)
exp(-21)2
(14)
The stationary distribution for Rord, neve, Runit is given by the stationary distribution and hence (1 - (x))//((x) S S. Taking roots now yields
<tt) S supl Qr Q20221TV S VSeep(-1Ax((R)1))
(15)
a.a
Finally, we conclude as in Schröder al. (2023):
log QiPdata log Qvpo(y)
KL(QQPaata Qipe)
Σ
log (og) QiPdata(y)
px(x)
yEX
24 +
Σ
U (l..o - U2( llo (x)
yEN
with U independent entropy term āe = E Q (2) log QePdata (v). After taking expectations with respect to Pdata (2) we find
0 < KL(Qipanta Qipp)
= 24 + Σ (l)
Σ Uo(rr)paata()) Σ
log po (2 (3)) (x)
VEX
#EX
TEX
ED (pdata U8) [xsses(s) [log pe() S c(t) (L || p8)
A.2 Eigenvalue Decomposition of Rate Matrices for Proposition
The rate matrices for the cyclical and for the ordinal graph structures have similar structure and are referred to as circulant and tridiagonal matrices. The easiest method for deriving the eigenvalue decompositions consists in deriving recurrence relations for the characteristic polynomial. systematic study of block circulant matrices can be found in Tee (2007) and a study of tridiagonal matrices was given in Losonczi (1992); Yuch (2005). These more general results may be helpful when constructing perturbations for spaces with a more complex structure than the ones introduced in this work. We take already existing results and check that the desired results hold.
Proposition 1. Assume the density qe (bla) = qe(y b/x a) is defined by the rate matrices Reye or Rord The transition density for all a, b 21, S} is given by
eve (vo(b
Σ
exp(2ribuuyy) exp ((2 (220(222 2)t) exp(- 1ap(-22iin (2ti-ii
(8)
5
p=1
dt
ard (b(a)
2
Σ
1
cos((2b 1)T exp ((2 cos 2))t cos((2a 1)T grd) (9)
Zp
p=1
where w (p - 1)/8 and word 11)/28, respectively, and zp (2, 1, 1,...,11 1).
Proof. In the circular case, the identity Rvp Apv reduces for single row to
Up Uppa+1 2úp Apup.a
(16)
16

---

For Up. = exp(-22Tip - 1aa/S)/v/ we find after dividing both sides by Up.a:
-2Ti (p- 1) 1) - (p 1)a ) exp (-2mi (p 1) + 1) - (p 1)a
Ap exp
-2
5
(17)
2n P -
= 2cos
-2
S
Furthermore, it is known that the Fourier basis is unitary, i.c. we have for a, b E 21.
s Σ
s
2wi (a () 1) 8(a,b).
Ub. Ub
Σ p=1
exp
(18)
In the tridiagonal case, we have to study two cases. Firstly, we have to check in row 1 using 2 (x) cy) cos(x + y) cos(x y):
2n (p 1)
T
T 1)
2p Up, ApUp,1 coos
COS
cOs
(19)
25
25
25
(p 1)
Tpp 1)
2 (p
1)
(pp 1)
<<(p 1)
cos
+
cos
ccos
25
28
25
28
25
(3 1)
(p
1)
cos
cos
25
25
and equivalently for row S. In all other rows have Up Up.a-1 20p.a ApUp. from
(2a ()(p 10--1 1)
2(p 1)
(2(a = 1) 1)(p 1)
(20)
25
25
2S
A.3 Proof of Scaling limit in Theorem 2
It is a typical generalisation of the central limit theorem that random walks attain Brownian motion as a universal scaling limit. We reproduce similar arguments for the law of the continuous time Markov chain. Without loss of generality we shift the state space by one and consider the state space (0, ........ S - 1} with eyclical and ordinal structure and let y/1 (-(x μ.S), where we always assume that the process is initialised at state J.S. Furthermore, we introduce the process 2, which an unconstrained continuous time Markov chain on with rate matrix Rao = -2, Ra,a+1 = 1, R = 1. The constrained process can then be described in terms of the unconstrained one: Let ps Z - {0, I.... ,,, 1} with yps (2n. + p) = p and (ps((2n + 1)S + p) = -p for for S S- and n E Σ. Then, ips reflects the unconstrained process ž₁ the boundaries 0 and S - i.e. yg = (2t) and y
E (0,1,
mod S. For the unconstrained process we define the holding times, i.e. the random time intervals which the process does not change state
h inf (t 2 F a) 120
(21)
and the jump process N. := Ense 8(2h # 2n-), where 2い- lim 2, which counts the number of state transitions up to time t. It is standard result that the holding times are exponentially distributed (Anderson, 2012, Proposition 2.8) ha Exp(-Raa). Furthermore, since all holding
times are identically distributed and Rao 2, the resulting jump process has Poisson distribution
N1 ~ Poisson(2t).
(22)
With these definitions, we can now first proof the Gaussian limit of 2 and then derive the limit of ye-
Theorem (Scaling limit). Let yt ~ q-(-1x = μs) with μ E 2/S,2 1}, where qe is either
the transition density of the cyclical or ordinal perturbation. Let y R (0, 1], where for all E Z
and x E (0, 1] g (n++a x) = x and ardd + x) = and (2n + 1+ x) = xx. Then,
8 S000 sp(Et) with & N(,,
17

---

Proof. First, we write 24 as the sum of independent increments starting at I = μ.S.
"t = S +
J3 p(Js 1) -( =/2..
(23)
We can now compute the characteristic function xs((s) = Elexp(iszz)) of rescaled in space and time. We have the following:
(s(()) E
exp
is
!sat
(24)
S
N
exp(isp) E
E
exp
J
Ngat
j=1
Nga.
cxp((isu)BB
II
Ngat
5
j=l
8
K
(2S²t)* K exp(-252t)
exp(isp) exp (25 cos (3) 25²r)
= exx((isp)
cos
S
K!
K=0
where we used that (os(x) = (ex(xxx( + (ap(-iz)) in the third step, the Poisson distribution of
Nr in the fourth step, and the series expansion of the exponential function in the final step. Since cos(r) x 1- 1/2x², we now have point-wise for any and due to the fact that 8/S + 0
(s(s) S exp(isp 1.s2)
(25)
which is the characteristic function of Gaussian with variance 2 and mean μ.. This proves the convergence in distribution of the rescaled unconstrained process 2,, Furthermore, for yord and tv it holds p = p(z/ /8) for all S E N. Furthermore, soord, eove are continuous maps from I to j0, 1) with reflecting or periodic boundary conditions. We thus have by the continuous mapping theorem
Usa S
ps((ga))
S-+00 (Et)
(26)
S
5
with & (u, 2t).
B Estimation of Energy Discrepancy
The energy discrepancy functional takes the form
EDy(Pasta: U) = Ex (x)) (U(x) Epenca (x) Bxty
(27)
for a conditional perturbing density and contrastive potential
Ug(y) log Σ exxxxx'))x((U(xx)) X'EX
(28)
While in theory energy discrepancy yields valid training functional for energy-based models for almost any choice of conditional distribution q, the conditional distribution also needs to allow an estimation of U, with low variance. This is particularly easy when q is symmetric, i.e. q(y/x) q(xly) in which case the contrastive potential can be expressed as an expectation which can readily be approximated from samples. This leads to
N log
M
(U) = N
W
+
log(MM
(29)
with x ~ Pdata. y ~ qe (- and is qe(-l (y-), where the offset w stabilises the loss approxima- tion discussed in Schröder et (2023). The interpretation of w is that contributions from negative
samples with U(x > (x+) are exponentially suppressed as contributions to the loss functional, thus avoiding the energies of negative samples to explode.
18

---

6.00
0.25
1.00
6.50
0.75
1.50
0.02
1.50
100
500
0.25
0.50
0.75
1.50
0.00 0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.50
6/00
0.45
0.50
0.75
1.00
0.02
0.25
0.58
9.75
L.00
0.55
0.50
0.5
0.50
S
5 20
= 50
5 100
5 500
-25
3.0
05
1.5
1.5
-2
-15 0.0
2.3
7.5
-5
10
15
-
5 20
50
5 100
- 500
-0.5 0. 05 15
0.0
7.5
-5
20
15
-0.5
1.0
1.
75
Figure 5: Scaling limit of the introduced perturbations. Top: Convergence of rescaled cyclical and ordinal perturbations S for base time parameters t 0.01 and t 0.05 to Gaussian [0, 1) with non-trivin trivial boundary conditions. One can see that the perturbation converges to a fixed shape on the normalised state space. Bottom: Convergence of rescaled cyclical and ordinal perturbation (yst E Eyys())/ /S for base time parameters 0.1 and 0.5 to Gaussian R (red line). The orange mark indicates the initial state. One can see that the perturbation remains non-trivial as the state space grows to infinity at rate VS.
19

---

B.1
Binary Case
On binary spaces, the construction of perturbations is particularly simple. We give some details in this subsection.
Bernoulli perturbation. As proposed previously in (Schröder et al., 2023, Appendix B.3), y can be defined as a Bernoulli distribution. Specifically, for & ~ BernoulliE)) E (0, 1) define the Bernoulli perturbed data point as y x x & mod 2. This induces a symmetric transition density q(y)x) on {0, 13". The Bernoulli random variable & emulates an indicator function, signifying in each dimension whether to flip the entry xf The value of controls the information loss induced by the perturbation. In theory, larger values of c lead to more Kata-efficient loss, while smaller values of may be more practical as they contribute improved training stability.
It is easy to compute the matrix exponential
exp t (1(+-))) -
11 -
1 1 e-
1/2 1/2 1/2
1/2
1- 1 1 1 e
(30)
thus relating the continuous time Markov chain framework on (0,1) Bernoulli perturbation with parameter 0.5 (1- -ee"" ee)
Neighbourhood-basee bbased perturbation and grid perturbation. Inspired by concrete score matching (Meng et al., 2022), one can introduce a perturbation scheme based on neighbourhood maps: x -> N(x), which assigns each data point x E X' a set of neighbours N Nxx). In this case, the forward transition density given by the uniform distribution over the set of neighbours, i.e. g(ylx) WX] (G) (y). A special case is the grid neighbourhood
Ngria (x) = (y E {0, 17" y y tcg, 1.2....,d! dd.
(31)
where eş is vector of zeros with a one in the k-th entry. Notably, this neighbourhood structure also exhibits symmetry, i.e., AVgila (x) Ngrria The same perturbation can be derived from Euler discretisation of the continuous time Markov chain. On binary space we have for t 1 for the same rate matrix as in Equation (30)
F = exp(() R idd+ (1 - !)
(32)
which is a completely deterministic perturbation which always changes the state of the data point. Combining this with the localisation grid find
Ed
Σ Fyu = [1/d y x Fe:
{1od
((ylx)
(33)
else
thus recovering the grid perturbation.
B.2 Connection to pseudo-likelihood estimation
Define q(vlx) Σ 5(yk, ()o(y x-k) which masks exactly one entry of the data vector. For simiplicity, we write y = x for the masked state and ommit the D. Then, energy discrepancy given for sampled perturbation y *
Up(x) log Σ g(y = x-1lx) exp(-
(34)
XEX
exp(-Uo(x))
lloo ExE(1.2....)) expl (U, 8, ....) log Po Po x-n)
Hence, this specific ED loss function is indeed Monte Carlo approximation of pseudo- likeldood Energy diserepancy offers additional flexibility through the tunable choice of and M, thus making ED adaptable to the structure of the underlying space and more efficient in practice, since the normalisation the ppeudo-likelihood only computed from M samples and does not require the integration along entire state space dimension.
20

---

Algorithm Training
Algorithm 2 Sampling
perturb the training sample x using (35)
1: initialise samples
y (e(+)x)
= ~N( I); x 0d Uniform(S2
2: for ...... do
2: generate M negative samples
for kt- deat
x ~ ++((-l),, i= 1, .......... M
4: update numerical features using (36)
3: evaluate the energy difference
x x <-x (x)( Vew, w w~
dø +- M Ei= exp(Us(x) - Ux(x'))
5: update categorical features using (37)
xt + zcat pa (2 (x )
4: update parameter 0 using (29)
end for
Va log(w/M de)
7: end for
The training and sampling procedures of energy discrepancy on tabular data. We use one training sample only to illustrate.
Figure 6:
C Tabular Data Synthesising with Energy-Basee Models
In this section, we introduce how to use energy discrepancy for training an energy- based model on tabular data. Let dnum and deat be the number of numerical columns and categorical columns, respectively. Each row in the table is a data point represented as a vector numerical features k== (1, ( To train an EBM with energy discrepancy. one should define the perturbation methods, which can be done by solving the differential equation in (6). For the numerical features, we choose the Gaussian perturbation as Schröder al. (2023), which has the transition probability in the form
and categorical features x = [x_um x=**], where xum E Rdom and x E 8
Qt m mum |x mum) Ny mum f1).
For the categorical features, we perturb each attribution independently:
deat
cat (y
cut
cat)
II
cot
k=1
Accordingly, there are three different perturbation methods for qe (2 12) Uniform perturbation defined in (7):
1 +
lt
(Uk
cat a
cat cat 1 = - -55
8(y cat a)
Here, the time parameter rescaled to be independent of S for ease of implementation. Cyclical perturbation defined in (8):
(v
cat
exp(2rinff* (rr) w00 exp (2 - 2)t) exp(-2miizs 2 wyc)
S
Ordinal perturbation defined in (9):
5
2
ord
(v)
cut
k
cut
(2t/ - 1m) exp (2Tw - 2)() cos((2rr) - 1) ord
S
Zp
To reduce the scale of noise, we further introduce grid perturbation, which involves perturbing only attribute time
dest
af catat catt
qe (v
deat
Theoretically, grid perturbation can be used alongside any type of perturbation described in (7, 8, 9). In our experimental studies, only explore the combination of grid perturbation with uniform
21

---

Figure 7: Additional density estimation results on the dataset with 16 dimensions and states.
Figure 8: Additional density estimation results on the dataset with 12 dimensions and 10 sates.
perturbation. By combining the Gaussian perturbation and categorical perturbation together, we can then draw the negative samples x_ via y (=(-)x), and x (-(-ly), where
(e(y|x) = Giua (y num x mmm you (orat|xeatt
(35)
Therefore, the energy function Up can be learned by minimising the loss function (29). We summarise the training procedure in Algorithm 1.
After training, new tabular data is synthesised by alternately applying Langevin dynamics and Gibbs sampling. Specifically, we update the numerical feature x hum via Langevin dynamics
num
x mum
V Xxmam (ox
x x + Vew, N(0.
(36)
For the categorical feature cat, we employ Gibbs sampler
k
Pa (x num xcat)
20 cat xx))))
k= 1,2, deat:
(37)
where cat denotes the vector cat excluding the k-th attribute. The sampling procedure is sum- marised in Algorithm 2.
22

---

Table 3: Experimental results of discrete density estimation. We display the negative log- oog-likeldhodd (NLL). The results of baselines are taken from Zhang et al. (2022a).
Metric
Method
2spirals
8gaussians
circles
moons
pinwheel
swissroll
checkerboard
PCD
20.094
19,991
20.565
19.763
19.593
20.172
21.214
ALOE+
20.062
19.984
20.570
19.743
19.576
20.170
21.142
EB IB-GFN
20.050
19.982
20.546
19.732
19.554
20.146
20,696
NLL1
ED-Bem 20.039
19.992
20.601
19.710
19.568
20.084
20.679
ED-Grid
20.049
19.965
20.601
19.715
19.564
20.088
20.678
Table 4: Experimental results of discrete density estimation. We display the MMD (in units of 1 x 10- The results of baselines are taken from Zhang et al. (2022a).
Metric
Method
2spirals
8gaussians
circles
moons
pinwheel
swissroll
checkerboard
PCD
2.160
0.951
0.188
0.962
0.505
1.382
2.831
+LOE+
0.149
0.078
0,636
0.516
1.746
0.718
12.188
EB 3F-GFN
0.583
0.531
0.305
0.121
0.492
0.274
1.206
MMDJ
FD-Berm ED-Grid
0.120
0.014
0.137
-0.088
0.046
0.045
1.541
0.097
-0.066
0.022
0.018
0.351
0.097
2.049
D Additional Experimental Results
In this section, we present the detailed experimental settings and additional results. All experiments are conducted on a single Nvidia RTX 3090 GPU with 24GB of VRAM.
D.1 Discrete Density Estimation
Experimental Details. This experiment keeps a consistent setting with Dai et al. (2020). We first generate 2D foating-points from a continuous distribution 6 which lacks closed form but can be casily sampled. Then, each sample x := [$1, X2] E point E {0, 1] using Gray code. To be specific, given & n. we quantise both & and &2 into 16-bits binary representations via Gray code (Gray, 1953), and concatenate them together to obtain a 32-bits vector As a result, the probabilistic mass function in the discrete space is p(x) & á ([Gray (1.1)) Gray (o To extend datasets beyond binary cases, we adhere the same protocol but utilise base transformation instead. This transformation enables the conversion of floating-point coordinates into discrete variables with different state sizes. It is important highlight that learning EBMs in such discrete spaces presents challenges due to the highly non-linear characteristics of both the Gray code and base transformation.
I22 is converted to a discrete data
We parameterise the energy function using a 4 layer MLP with 256 hidden dimensions and Swish (Ramachandran et al., 2017) activation. To train the EBM, we adopt the Adam optimiser with a learning rate of 0.0001 and batch size of 128 to update the parameter for 10 steps. For the energy discrepancy, we choose w = 1, M = 32 and the grid perturbation for all variants. For contrastive divergence, we employ short-run MCMC using Gibbs sampling with 10 rounds (i.e., 10 S steps).
After training, we quantitatively evaluate all methods using the negative log-likelihood (NLL) and the maximum mean discrepancy (MMD). To be specific, the NI.I. metric is computed based on 4, 000 samples drawn from the data distribution, and the normalisation constant is estimated using importance sampling with 1, 000, 000 samples drawn from variational Bernoulli distribution with p = 0.5. For the MMD metric, we follow the setting in Zhang et al. (2022a), which adopts the exponential Hamming kernel with 0.1 bandwidth. Moreover, the reported performances are averaged over 10 repeated estimations, each with 4, 000 samples, which are drawn from the learned energy function via Gibbs sampling.
Qualitative Results. In Figures 7 and 8, we present additional qualitative results the learned energy on datasets with 5 and 10 states. We see that ED consistently yields more accurate energy landscapes compared CD. Notably, we only showease results using grid perturbation with the uniform rate matrix, as qualitative findings are consistent across different perturbation methods. Additionally,
23

---

Table 5: Statistics of the real-world datasets.
Dataset
# Rows
# Num
# Cat
# Train
Validation
# Test
Task Type
Adult
48, 842
6
28, 943
3, 618
16, 281
Binary Classification
Bank
45,211 211
10
36, 168
4, 1,521
4, 4,522
Binary Classification
Cardio
70, 000
5
6
56, 000
7,000
7,000
Binary Classification
Churn
7. 043
3
15
5, 634
704
705
Binary Classification
Mushroom
61, 096
17
48, 855
107
6, 107
Binary Classification
Beijing
834
7
5
35, 058
4,383
4,383
Regression
Peporret
Covered
Copereed OABEMEC
Conmed CETSeRM
Coperne Collidder Ere
1003000
Figure 9:
Comparison of calibration results between the baseline (top) and energy discrepancy (bottom) on varying datasets. Left to right: Bank, Cardio, Churn, Mushroom.
empirically observe that gradient-based Gibbs sampling methods (Grathwohl et al., 2021; Zhang al., 2022b) tend to generate samples outside the data support more readily. In this regard, we only display the results of CD methods using vanilla Gibbs sampling.
Quantitative Results. The quantitative results are illustrated in Tables 3 and indicating the superior performance of our approaches in most scenarios. Notably, previous studies on discrete EBM modelling exclusively focus binary cases. As result, we only present the quantitative comparison for the dataset with 2 states.
D.2 Tabular Data Synthesising
Experimental Details for the Synthetic Dataset. For the synthetic dataset, we parametrise the energy function using three MLP layers with 256 hidden states and Swish activation. To handle mixed data types, we transform each categorical feature into a 4-dimensional embedding using linear layer, and then concatenate these embeddings with the numerical features as input. To train the EBM, we apply the Adam optimiser with learning rate of 0.0001 and batch size of 128. We update the parameters over 1, 000 epochs, with each epoch consisting of 100 update iterations. For ED, we set w 1, MM 32, using Gaussian perturbation for the numerical features and grid perturbation for the categorical features. For CD, we incorporate the replay buffer strategy and employ Langevin dynamics and Gibbs sampling with 50 rounds (totalling 50 + S steps) for the numerical and categorical features, respectively.
Experimental Details for the Real-world Dataset. Table 5 summarises the statistical properties of the datasets. To parameterise the energy function and handle mixed data types, we use the same approach but with 1024 hidden units instead of 256. We train the model using the AdamW optimiser (Loshchilov & Hutter, 2019) with a learning rate of 0.0001 and a weight decay rate of 0. 0.0005. The model is trained for 20, 000 update steps with a batch size of 4096. For ED, Gaussian perturbation is employed for numerical features, while categorical features undergo different perturbation methods. Specifically, Uni and TabED-Grid use uniform and grid perturbations with / 0.1, respec- tively. For Tab and TabED-Ord, corresponding to cyclical and ordinal perturbations, quadratic scaling is applied with t chosen from the best performance in {0.01, 0.005, 0.001 Morcover, CD
24

---

Table 6: Results on density similarity between the synthesis and real tabular data.
Single-column Density Similarity 1
Pair-wise Correlation Similarity
Methods
Adult
Bank
Cardio
Churn Mushroom
Beijing
Adult
Bank
Cardio
Chum Mushroom
Beijing
CTGAN
.844
.901
.799
.744
.769
.717
.826
.761
TVAE
.783
-824
.899
.965
.711
.669
.772
.687
.808
.919
.688
TabCD
.719
.790
.824
.855
.618
.799
.522
.600
.629
.710
.488
.761
TabDDPM
.98
.998
.992
.76
.987
.980
.975
.894
.870
.953
.962
.946
TabED-Uni
.785
.779
.914
.886
.888
.933
.653
.683
.783
.808
.770
.793
TabED-Grid
.751
.766
.945
.846
.951
.583
.768
.764
.842
.842
TabED-Cye
.778
.834
.751
.636
.703
.755
6885
TabED-Ord
.288
.894
.887
.747
.702
.796
.791
.826
.662
TabED-Str
.77
.798
.892
.632
.660
.759
Q
6
7
8
3
2
2. 38
2
5
9
3
4
2
5
0
9
9
9
2
7
F
0
5
8
4
Q
2
9
1
1
3
8
1
2
0
2
4
9
9
8
4
6
7
3 8 8 1 0
7
4
1 4
1 2 1
1
4
7
0 8
9
2 0 0 2
8
4
1
(
6
0
3 0 9
5 6
2
3
D
1/
0
2
2
5
&
9
1
4
4
6
9
4
2
1
4
4
8
4
7
7
2
7
0
2
R
3
4
0
5
8
0
6
0
4
0
8
8
0
3
8
2
1
2
3
6
6
8
Z 6
X
8
4
9
3
8
7
9
9
8
F
7
3
E 9 5
0
/
5
8
Ç
6
2
2
3
1
9
8
5
4
7
7
5
9 7
5 y 2
4
J
2
X
0
3
2
A
7
/
3
0
8
2
8
9
0
8
2.
8
V
5
S
0
R
2
2 9
0
6
9
0
2
3
6
9 9
0
6
9
O
8
6
Figure 10: Dynamic MNIST samples generated from the learned energy function using ED-Grid with various sampling methods. Left to right: GWG, GFlowNet, GFlowNet+GWG.
utilises the same algorithm in the synthetic dataset, but with 10 steps for short-run MCMC. The reported results are averaged over 10 randomly sampled synthetic data.
Experimental Details for Calibration. Let y and x be the target label and the rest features in the tabular data, we can transform a learned EBM (x. y) into a deterministic classifier: (y)M(y()) oc exp(-Uo(X, 7)). As baseline for comparison, same architecture by maximising the conditional likelihood: Epaci [log PCLF (3) x)]. In particular, we utilise the Adam optimiser with learning rate of 0.001 and batch size of 4096 to train the classifier PeLF. The model undergoes training for 50 epochs.
additionally train classifier PCLe(y/ (3/x) with the
Additional Results for Calibration. Figure 9 presents additional calibration results across different datasets. It shows that the energy-based classifier learned by energy discrepancy exhibits superior calibration compared to the deterministic classifier, except for the Mushroom dataset, where the deterministic classifier achieves 100% accuracy, resulting low calibration error.
Additional Results with Other Metrics. We evaluate our methods against baselines using two additional metrics: single-column density similarity and pair-wise correlation similarity. These metrics assess the similarity in the empirical distribution of individual columns and the correlations between pairs of columns in the generated versus real tables. Both metries can be computed using the open-source SDMetrics API. As shown in Table 6, the result shows that the proposed ED-based approaches either outperform or achieve comparable performance the baselines across most datasets.
D.3 Discrete Image Modelling
Experimental Details. In this experiment, we parametrise the energy function using ResNet (He et al., 2016) following the settings in Grathwohl et al. (2021); Zhang et al. (2022b), where the network has residual blocks with 64 feature maps. Each residual block has 2 convolutional layers and uses Swish activation function (Ramachandran et al., 2017). 1). We choose M = 32, w 1 for all variants of energy discrepancy, 0.001 in Bernoulli perturbations. Note that here we choose relatively small since empirically find that the loss of energy discrepancy converges to constant rapidly with
25

---

Table Running time complexity comparison for energy discrepancy and contrastive divergence.
Time
Method
CD-1
CD-5
CD-10
ED-Bem
ED-Grid
Per Iteration (s)
0.0583
0,1904
0.3351
0.0905
0.0872
Per Epoch (s)
29.1660
95.2178
167.5718
46,4317
44.0621
Table 8: Experimental results of the comparison between energy discrepancy and contrastive diver- gence with varying MCMC steps.
Dataset') \\ Method
CD-1
CD-3
CD-5
CD-7
CD-10
ED-Bern ED-Grid
Static MNIST
182.53
130.94
102.70
98.07
88.13
96.11
90.61
Dynamic MNIST
157.14
130.56
97.50
91.00
84.16
97.12
90.19
Omniglot
nan.
161.96
142.91
149.68
146.11
97.57
93.94
larger which can not provide meaningful gradient information to update the parameters. All models are trained with Adam optimiser with a learning rate of 0.0001 and a batch size of 100 for 50, 000 iterations. We perform model evaluation every 5, 000 iteration by conducting Annealed Importance Sampling (AIS) with the GWG (Grathwohl et al., 2021) sampler for 10, 000 steps. The reported results are obtained from the model that achieves the best performance on the validation set. After training, we finally report the negative log- ligeliiel by running 300, 000 iterations of AIS.
To qualitatively assess the validity of the learned EBM, this study presents generated samples from the dynamic MNIST dataset. We first train an EBM using ED-Grid and then synthesise samples by employing various sampling methods, including: i) GWG (Grathwohl et al., 2021) with 1000 steps: ii) GFlowNet with the same architecture and training procedure as per Zhang et al. (2022a); and iïi) GFlowNet followed by GWG with 100 steps.
Qualitative Results.
Empirically, we find that the quality of generated samples can be improved with more advanced sampling approaches. As depicted in Figure 10, the GWG sampler suffers from mode collapse, leading to samples with similar patterns. In other hands, GFlowNet enhances the quality to some extent, but it produces noisy images. To address this issue, we apply GWG with 100 steps following the GelowNet. can be seen that the resulting GFlowNett+GGG sampler yields the highest quality with clear digits. These observations validate the capability of our energy discrepancies to accurately learn the energy landscape from high-dimensional datasets. We leave the development of a more advanced sampler in future work to further improve the quality of generated images using our energy discrepancy approaches.
Time Complexity Comparison for Energy Discrepancy and Contrastive Divergence. Energy discrepancy offers greater training efficiency than contrastive divergence, as does not rely on MCMC sampling. In this experiment, we evaluate the running time per iteration and epoch for energy discrepancy and contrastive divergence in training a discrete EBM on the static MNIST dataset. The experiments include contrastive divergence with varying MCMC steps and variants of energy discrepancy with fixed value of M = 32. The results, presented in Table 7, highlight that ED-Bern and ED-Grid are the fastest options, as they do not involve gradient computations during training.
Comparison to Contrastive Divergence with Different MCMC Steps. Considering the greater training efficiency of energy discrepancy over contrastive divergence, this study comprehensively compares these two methods with varying MCMC steps in contrastive divergence. Specifically, we utilise the officially open-sourced implementation" of DULA to conduct contrastive divergence training. As depicted in Table 8, we find that energy discrepancy significantly outperforms contrastive divergence when employing single MCMC step, and achieves performance comparable to CD-10. We attribute this superiority to the fact that CD-1 involves a biased estimation of the log-likelihood gradient due to inherent issues with non-convergent MCMC processes. In contrast, energy discrepancy does not suffer from this issue due to its consistent approximation.
The Efficacy of the Number of Negative Samples. In all experiments, we choose the number of negative samples as M 32 irrespective of the dimension of the prob- lem, to maximise computational efficiency within the constraints of our GPU capacity.
26

---

To investigate the impact of the number of nega- tive samples on performance, we conduct exper- iments by training energy-base models on the static MNIST dataset with ED-Grrd for different values of M. As detailed in Table 9, our results maintain comparable quality even as the number of negative samples is decreased. Notably, our approach offers greater parallelisation potential compared to the sequentially computed MCMC of contrastive divergence.
Table 9: Discrete image modelling results of ED-Grid on the static MNIST dataset with differ-
M and w
M
M 16
M 32
NLL 90.13
90.37
89.14
90.61
D.4 Training Ising Models
Task Descriptions. We further evaluate our methods for training the lattice Ising model, which has the form of
xxx exp(x" Jx), (
Figure 11: Results on learning Ising models. Left to right: ground truth, ED-Bem, ED-Grid.
where J = AA with E R and Ap being the adjacency matrix of Grathwohl et al. (2021); Zhang et al. (2022b,a), we generate training data through Gibbs sampling and use the generated data to fit a symmetric matrix J via energy discrepancy. Note that the training algorithms do not have access to the data Gata-generating matrix J, only to the collection of samples.
D grid. Following
Experimental Details. As in Grathwohl et al. (2021); Zhang et al. (2022a,b), we train a learnable connectivity matrix Jg estimate the true matrix J in the Ising model. To generate the training data, we simulate Gibbs sampling with 1, 000, 000 steps for each instance to construct a dataset of 2, 000 samples. For energy discrepancy, we choose w M 32 for all variants, E 0.1 in Bernoulli perturbations. The parameter Jø is learned by the Adam (Kingma & Ba, 2015) optimiser with learning rate of 0.0001 and a batch size of 256. Following Zhang et al. (2022a), all models are trained with an I regularisation with coefficient in (100, 50, 10, 5, 1, 0.1, 0.01) } to encourage sparsity. The other setting is basically the same as Section F.2 in Grathwohl al. (2021). We report the best result for each setting using the same hyperparameter searching protocol for all methods.
Qualitative Results. In Figure 11, consider D 10 10 grids with 0.2 and illustrate the learned matrix J using a heatmap. It can be seen that the variants of energy discrepancy can identify the pattern of the ground truth, confirming the effectiveness of methods.
Quantitative Results. In the quan- titative comparison to the baselines, we consider = 10 x 10 grids with :
Table 10:
Mean negative log-RMSE (higher is better) be- tween the learned connectivity matrix Jo and the true matrix for different values of D and o. The results of baselines are directly taken from Zhang et al. (2022a).
0.1, 0.2.0........ ... 0.5 and D 9 grids with ` = 0, -0.2. The methods evaluated by computing the negative log-RMSE between the estimated Jo and the matrix J. As shown in Table 10, our methods demonstrate comparable results the baselines and, in certain settings, even outperform Gibbs and GWG, indicat- ing that energy discrepancy is able to discover the underlying structure within the data.
D
10
D
ga
Method a
0.1
0.2
0.3
0.4
0.5
-0.1
-0.
Gibbs GWG
4.8
4.7
3.4
2.6
2.3
4.8
4.7
4.8
4.7
3.4
2.6
4.7
EB-GEN
6.1
5.1
3.3
2.6
2.3
5.7
5.1
ED-Bern ED-Grid
5.1
4.0
2.9
2.5
2.3
5.1
4.3
4.6
4.0
3.1
2.6
4.5
4.0
D.5
Graph Generation
Task Descriptions. The efficacy of our methods can be further demonstrated by producing high- quality graph samples. Following the setting in You et al. (2018), our model evaluated on the Ego-small dataset, which comprises one-hop ego graphs extracted from the Citeseer network (Sen
27

---

generate new (a) Training Data
(b) ED-Bem
(c) ED-Grid
Figure 12: Visualisation of the training data and samples drawn from the energy based models learned by the variants of our approaches on the Ego-small dataset.
et al., 2008). We consider the following baselines? in graph generation, including Graph (Simonovsky & Komodakis, 2018), DeepGMG (Li et al., 2018), GraphRNN (You al., 2018), GNF (Liu et al., 2019), GrappAF (Shi et al., 2020), GraphDF (Luo et al., 2021), EDP-GNN (Niu et al., 2020), RMwGGIS (Liu et al., 2023), and contrastive divergence with GWG sampler (Grathwohl et al., 2021).
Experimental Details. Following the setup in You et al. (2018), we split the Ego-small dataset, allo- cating 80% for training and the remaining 20% for
Table 11: Graph generation results in terms of MMD. Avg. denotes the average over three MMD results.
testing. To provide better insight into this task, we illustrate subset of training data in Figure 12a. No- tably, these training data examples closely resemble realistic one-hop ego graphs.
Method
Degree
Cluster
Orbit
Avg.
GraphVAE
0.130
0.170
0.050
0.111
DeepGMG
0.040
0.100
0.020
0..53
For a fair comparison, we parametrise the energy function via 5-layer GCN (Kipf & Welling, 2017) with the ReLU activation and 16 hidden states for all energy-based approaches. For hyperparameters, choose M = 32, w = 1 for variants of energy dis- crepancy and 0.1 for the Bernoulli perturbation. Following the configuration in Liu et al. (2023), we apply the advanced version of RMWGGIS with the number of samples s = 50 (Liu et al., 2023, Equation 11). Regarding the EBM (GWG) baseline, we train it using persistent contrastive divergence with buffer size of 200 samples and the MCMC steps being 50. To train the models, we use the Adam optimiser with learning rate of 0.0001 and batch size of 200. After training, graphs by first sampling N, which is the number of nodes to be generated, from the empirical distribution of the number of nodes in the training dataset, and then applying the GWG sampler (Grathwohl et al., 2021) with 50 MCMC steps from randomly initialised Bernoulli noise. To assess the quality of these samples, we employ the MMD metric, evaluating it across three graph statisties, i.e., degrees, clustering coefficients, and orbit counts. Following the evaluation scheme in Liu et al. (2019), We trained 5 separate models of each type and performed trials per model, then averaged the result over 15 runs.
GrapRRNN
0.090
0.220
0.003
0.104
GNI
0.030
0.100
0.001
0.044
GraphAF
0.110
0.001
0.047
GraphDF
0.040
0.130
0.010
0.060
EDP-GNN
0.052
0.093
0.007
0.050
EBM (GWG)
0.095
0.061
0.032
0.063
RMwGGIS
0.066
0.042
0.036
0.048
ED-Bem ED-Grid
0.063
0.054
0.014
0l.44
0.036
0.050
0.019
0.035
Qualitative Results. We provide a visualisation of generated graphs from variants of methods in Figures 12b and 12c Notably, the majority of these generated graphs resemble one-hop ego graphs, illustrating their adherence to the graph characteristics in the training data.
Quantitative Results. In Table 11, we compare our methods to various baselines. can be seen that methods outperform most baselines in terms of the average of the three MMD metrics, indicating the faithful energy landscapes learned by the energy discrepancy approaches.
"There is insufficient information to reproduce EBM (GwG) and RMwGGIS precisely from Liu et al. (2023). We reran these two baselines with controlled hyperparameters for fair comparison, while other baseline results were taken from their original papers.
28

---

E Naming Conventions and Parameters of Introduced Methods
This table summarises the naming conventions and available tuning parameters for all introduced methods. The structured perturbation TabED-Str uses different perturbations depending on the state space structure: On unstructured data, the uniform perturbation with tuning parammerrretter leat is used, while on ordinally and eyclically structured data the ordinal perturbations and cyclical perturbations are used, respectively, with tuning parameter /hase.
Table 12: Overview of all introduced energy discrepancy methods
Name
Space (Discrete component)
Perturbation (Discrete component)
Tuning Parameter
ED-Bem
{0, 13dd
kum Bern(s)
0.5(1 2()
ED-Grid
{0, 1}ad
ER=1 uk 02-2a/=1
None
TabED-Uni
8-1 [L, .l.......
S&
Il exp Dyn (Equation (7))
>
TabED-Grid
f1 S
km [0n ()ō(y x-))
None
TabED-Cye
f
I k exp(tx (tx p JeEN (Proposition 1)
t. = fass
TabED-Ord
8- (1,
Sp}
IT. exp(te (1 June (Proposition 1)
Ix S tbss
TabED-Str
Sk)
===1 exp(ta Rt Juae (Mixed)
Mixed
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our methodology describing the extension of energy discrepancy to discrete and mixed state spaces is given Section and Section 4, the proofs supporting our claims claims with experiments
are found in section Appendix A of the appendix. We support on discrete and mixed data in Section 6, demonstrating that the methodology generalises to settings of various types like categorical data, tabular data, and binary image data.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The main limitations are outlined in the conclusions section. To the best of our knowledge, the dominating factor in the performance of our method is the assumption that data can be assumed as independent samples of positive density Pdata > 0. This assumption is violated for practically all data sets, but the extent to which the support of Pdata smaller than the state space deteriorates performance, either because of pre- matura convergence or because the perturbation does not explore the state space efficiently. Guidelines:
Justification:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
30

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Proofs for our results be found in Appendix A. Due the nature of our results a proof sketch within the page limit was not feasible.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental details, regarding the network architecture and hyper- parameters, are given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
31

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is anonymised and given in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Seaiddes/ooeSubmisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: / httpss.cc/uublll wetiiiigggiee////eeeee Sttp:/oooeSSSiiseeeeeeeee wtt.suummmsssiooooooo..ooo ssioccolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The full details are provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The quality of generated synthetic tabular data was tested by comparing the quality of a classifier trained on real and synthetic data. The error bar reflects the standard deviation of the classifier metric based on ten independently generated synthetic data sets sampled from the learned model. Error bars for negative log-likelihoods were not feasible with our computational resources due to the computational cost of annealed importance sampling.
32

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources, as well as the required experimental runs, are detailed in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics stctps://neure cc
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and have considered the societal impact. The paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
33

---

Answer: [Yes]
Justification: The broader impact discussed as part of the conclusion. Our method can be used for data imputation in tabular data which can be used downstream applications to discriminate or exclude if used irresponsibly, e.g. on biased data. However, we believe that our method is less prone such applications than existing methods for data mining.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper demonstrates foundational research tested on publicly available datasets that were designed to test machine learning algorithms.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly acknowledge and cite all assets and resources used in the paper.
Guidelines:
34

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasatss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper introduces a training method for energy-based models and does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper introduces training method for energy-based models and does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
35

---

Answer: [NA]
Justification: The paper introduces a training method for energy-based models and does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
36


Reviews:
[{"review_id": "7SLStw2dxK", "reviewer": "Reviewer_9ZAB", "summary": ": The paper introduces a novel method for training energy-based models (EBMs) on discrete and mixed data using heat equations on structured spaces. This method employs the Energy Discrepancy (ED) loss function, which eliminates the need for Markov chain Monte Carlo (MCMC) by using graph-structured perturbations. The proposed approach is evaluated on several applications, including discrete density estimation, synthetic data generation, and calibrated classification, demonstrating significant improvements in training efficiency and model performance.", "questions": "See weakness", "limitations": "See weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper successfully extends the Energy Discrepancy method to discrete and mixed data with solid theoretical analysis, addressing the challenge of robust and fast sampling in these space. The designed experiments demonstrate the method's ability to accurately capture complex data structures and generate high-quality synthetic data, highlighting its practical applicability.", "weaknesses": "1.\tDespite the method's solid contributions and experimental design, the motivations behind each step and their presentations are not very clear, making it hard to follow. For instance, in Section 3.1, the paper discusses different structured and unstructured categorical values, introducing the four types {cyc, ord, unif, abs}. However, it is not clear why these specific structures are chosen. Are they meant to cover all categorical values comprehensively, or are they the most common in tabular data? Providing a clearer rationale would help readers understand the choices made.\n\n2.\tThe scalability of the proposed method in such scenarios is a significant concern. An analysis or discussion on how the method handles large categorical values would be beneficial. This could include potential modifications or considerations to ensure that the method remains efficient and practical when applied to datasets with large categorical variables. What’s more, I strongly recommend moving these algorithms from the appendix into the main body of the paper. This would make the paper easier to follow and more accessible to readers who need to understand the detailed workings of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your helpful comments.\n\n> Despite the method's solid contributions and experimental design, the motivations behind each step and their presentations are not very clear, making it hard to follow. For instance, in Section 3.1, the paper discusses different structured and unstructured categorical values, introducing the four types {cyc, ord, unif, abs}. However, it is not clear why these specific structures are chosen. Are they meant to cover all categorical values comprehensively, or are they the most common in tabular data? Providing a clearer rationale would help readers understand the choices made.\n> \n\nThank you for letting us know that the presentation is not clear in this point, so we would like to provide additional background on this topic: To create the Energy Discrepancy loss we have to introduce a noising process that produces contrastive negative samples which the model can learn from. In principle, the noising processes {unif, abs} already cover **all categorical values** and this is the approach taken for discrete diffusion models, see e.g.\n\n[1] Campbell et al. A Continuous Time Framework for Discrete Denoising Models, NeurIPS 2022\n\n[2] Kotelnikov et al. TabDDPM: Modelling Tabular Data with Diffusion Models, ICML 2023\n\n[3] Lou et al. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, ICML 2024\n\nIn this work, we observe that many discrete types of data have additional structure such as seasonality/periodicity (e.g. month of year) or ordering/grading (e.g. age, RGB pixel values, salary) that give us more fine-grained control over the noising process than the diffusions in [1, 2, 3]. This generalises previous diffusions while remaining **simulation-free**, i.e. we can compute and sample from the noising distribution for any $t$ in closed form.\n\nThe intuition for respecting the structure of the underlying discrete space is that we seek negative samples that are close to the data support to provide a good training signal (This is similar to the discriminator training in GANs).\n\nIf the discrete space of interest has a more complicated graph structure, the process either needs to be simulated from eigen decompositions of the graph Laplacian or one can resort to the simpler uniform or absorbing perturbation. Since all our benchmark data sets didn't require an involved graphical structure, we stick to the more common ordinal or cyclical structure which generalises previous noising processes [1, 2, 3].\n\n> The scalability of the proposed method in such scenarios is a significant concern. An analysis or discussion on how the method handles large categorical values would be beneficial. This could include potential modifications or considerations to ensure that the method remains efficient and practical when applied to datasets with large categorical variables.\n> \n\nThank you for your suggestions. Could you elaborate what you mean by large categorical values in case we misunderstand your question?\n\nScalability is indeed a valid concern for energy-based models in general. Most scalable **continuous** EBMs such as [4], which trains EBMs on image net 32x32, have other problems, e.g. over-smoothing the estimated densities due to missing theoretical guarantees, and are limited to the continuous domain. This limits their applicability in some key downstream tasks like reliable out of distribution detection.\n\nFor discrete data sets, alternative training methods for EBMs produce comparable results to our method. Compared to MCMC based training methods like contrastive divergence [5], the energy discrepancy approach requires less tuning and runs faster since the loss can be computed in a single parallel pass through the network and does not rely on sequential steps. Furthermore, these works have yet to be extended to mixed state spaces, and many tabular data sets of interest have a comparably small number of features.\n\nWe are also interested in better scalability in future work. However, this will likely require a combination of EBMs with other generative modelling approaches such as a variational decoder model as in [6]. This method has already been established for the energy discrepancy training method. Secondly, the energy function may be learned with the energy discrepancy loss at various noise scales as in [7], thus improving training stability and learning a sampler alongside the EBM. Finally, EBMs may be an interesting way to modify a pre-trained base model as in [8]. While ED can likely be used to modify [6, 7, 8], this is not the main contribution of our work.\n\n[4] Du et al. Implicit Generation and Generalization in Energy-Based Models, NeurIPS 2019\n\n[5] Grathwohl et al. Oops I took a Gradient: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions, ICML 2021\n\n[6] Pang et al. Learning Latent Space Energy-Based Prior Model, NeurIPS 2020\n\n[7] Gao et al. Learning Energy-Based Models by Diffusion Recovery Likelihood, ICLR 2021\n\n[8] Deng et al. Residual Energy-Based Models for Text Generation, ICLR 2020\n\n> What’s more, I strongly recommend moving these algorithms from the appendix into the main body of the paper. This would make the paper easier to follow and more accessible to readers who need to understand the detailed workings of the method.\n> \n\nThank you for this suggestion. We will include the algorithms in the main text for the revision of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHChQ9CxXT", "reviewer": "Reviewer_qwnm", "summary": "The paper proposes a suite of methods for training energy-based models for discrete and mixed data using the Energy Discrepancy loss, a recently proposed method for training EBMs. Compared to contrastive divergence, it does not require MCMC sampling from the model distribution during training, improving training efficiency. This is done by simulating a diffusion process on the discrete states and effectively using those noisier samples as the negative samples. The paper introduces a connection between the new method and maximum likelihood estimation, showing that energy discrepancy as applied to discrete state spaces can converge to the negative log-likelihood. In experiments, the new method behaves favourably compared to contrastive divergence-based methods on synthetic data sets, on average better than baselines on real-world tabular data sets, and comparably to many competing methods generation on discrete image modelling. An application of the trained EBM on classification and improving uncertainty quantification compared to a direct classifier is also shown.", "questions": "- How should I interpret the left side of the equation in Theorem 1, and the fact that the right side approaches zero with large enough t? How does this link ED to maximum likelihood, exactly? \n- What is Avg. Rank in Table 1?\n\nOverall, the paper seems like a solid contribution in advancing the training of this branch of energy-based generative models. However, I was not aware of ED before reading this paper, and am not very up-to-date on the most recent work on energy-based models. As such, I give tentatively a weak accept.", "limitations": "Addressed adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper proposes a relevant extension to recently published work. Especially Theorem 1 does not seem obvious, and the paper may open up the use of the Energy Discrepancy loss to a much wider variety of use-cases. \n- The method is also quite simple, and seems simple to implement. \n- The paper connections to recent work on discrete diffusion models, and proposes a variety of methods to estimate the energy discrepancy loss. \n- The results are good compared to standard contrastive divergence based methods\n- The paper is well written, and I found it easy enough to understand even without prior knowledge on the Energy Discrepancy method.", "weaknesses": "- As noted in the limitations, the application to data such as images seems to be challenging as the noisy negative samples may not give very useful training signal in this case. \n- Although the energy discrepancy method has already been proposed and published in previous work, I found the justification for the method slightly confusing while reading this paper. What is Theorem 1 exactly saying? (see questions) The loss also is, in practice, approximated with something slightly different than the proposed loss, which seems conceptually a bit confusing. However, this is not a major concern given that the base method has been proposed and published in previous work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and questions.\n\n> Although the energy discrepancy method has already been proposed and published in previous work, I found the justification for the method slightly confusing while reading this paper. What is Theorem 1 exactly saying? (see questions)\n> \n\n> How should I interpret the left side of the equation in Theorem 1, and the fact that the right side approaches zero with large enough t? How does this link ED to maximum likelihood, exactly?\n> \n\nWe want to address both your question and the weaknesses mentioned in your review. Theorem 1 is meant to guide the hyperparameter choice of $t$. Another way of stating Theorem 1 is as follows:\n\n$$\n\\mathrm{ED}{q_t}(p_\\mathrm{data}, U_\\theta) - \\mathcal L_{\\mathrm{MLE}}(\\theta) - z_t = \\mathcal O(e^{-\\lambda t})\n$$\n\nHere, $z_t$ is a constant **independent** of $\\theta$, so the optimisation landscapes of energy discrepancy estimation and maximum likelihood estimation in $\\theta$ align at an exponential rate, except for a shift $z_t$ which does not affect the optimisation.\n\nPractically speaking, this property implies that large time parameters yield losses with favourable statistical properties as the MLE has the lowest variance among all unbiased estimators. The most striking example of this is global mode awareness, i.e. as shown in prior work [1], energy discrepancy losses are capable of estimating the mixture weight distributions in multi-modal data sets while being more tractable than maximum likelihood estimation. However, this comes at the cost of higher variance of the loss function due to the sample approximation used to compute the loss function in practice.\n\nSince arbitrarily large $t$ are impractical, the parameter needs to be tuned. For this reason we study the opposite limit $t\\to 0$ in the following sections. Similar to the late stages of GAN training, our goal is that negative samples obtained from the noising process remain close to the data support to provide a good training signal. The infinitesimal analysis relates the properties of ED for $t\\to 0$ to the geometry of the underlying discrete space. The perturbed samples produced are then with highest probability the nearest neighbours of the data point in the graph. This makes it possible to get fine-grained control about the properties of the ED loss with a single time parameter.\n\nWhile these results guide our understanding of the influence and tuning of $t$ in the algorithm, Theorem 1 is not needed to justify ED as a loss function. Instead, the justification was given in prior work and is reiterated in lines 78-81: One can show that after convergence, for a sufficiently rich family of neural networks and arbitrary amounts of data, $p_\\theta(\\mathbf x) = p_\\mathrm{data}(\\mathbf x)$ which holds for all $t$.\n\n> The loss also is, in practice, approximated with something slightly different than the proposed loss, which seems conceptually a bit confusing. However, this is not a major concern given that the base method has been proposed and published in previous work.\n> \n\nThe loss used in practice can be justified by observing that for any $w>0$:\n\n$$\n\\lvert \\mathcal L_{w, M, t}(\\theta) - \\mathrm{ED}{q_t}(p_\\mathrm{data}, U_\\theta)\\rvert \\xrightarrow{M, N\\to\\infty} 0\n$$\n\nThus, the estimated loss function is asymptotically unbiased, and empirically we observe that the loss performs similarly to our expectations from the theory. It should be noted that most EBM training approaches need some form of approximation and stabilisation. MCMC-based training of deep EBMs usually use short run MCMC, introducing biases into the derived parameter update, and use a regulariser $\\mathcal L_{\\mathrm{reg}}(\\theta) = U_\\theta(\\mathbf x_+^i)^2 + U_\\theta(\\mathbf x_-^i)^2$ which alters the desired MLE loss. In comparison, ED is easier to justify, and indeed we observe better performance in most applications.\n\nTo further support our theoretical analysis empirically, we conduct discrete density estimation experiments with different $w,M$ (see Figure 2 in the attached PDF for illustration). Despite using varying $w$, our approach can still learn a faithful energy landscape using a sufficiently large $M$, verifying the theoretical analysis that $\\mathcal L_{w, M, t}(\\theta) \\rightarrow \\mathrm{ED}{q_t}(p_\\mathrm{data}, U_\\theta)$ when $M, N \\rightarrow \\infty$.\n\n> What is Avg. Rank in Table 1?\n> \n\nWe rank each of the 7 methods in table 1 according to their AUC scores from 1 to 7. The average rank is the average ranking of the method across the used data sets.\n\n[1] Schroeder et al. Energy Discrepancies: A Score independent Loss for Energy-Based Models, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1B7xDcHrjH", "reviewer": "Reviewer_FxWc", "summary": "This paper extends the Energy Discrepancies framework introduced by Schroder et al. to the setting of discrete data. In order to do this, the authors first describe ways to perturb discrete data by modeling the perturbation process as a CTMC. They describe suitable perturbation choices for different types of discrete data (e.g. binary, categorical, ordinal, cyclical) and describe different considerations for the time parameter in the CTMC. They then propose an approach that performs a Monte-Carlo estimate of the contrastive potential needed for the Energy Discrepencies loss and compare their method to existing methods for training discrete EBMs.", "questions": "In section 3 on lines 93-95 the authors describe two key criteria for defining useful perturbation processes. The first criterion is described as “the negative samples obtained through $q$ are informative for training the EBM when only finite amounts of data are available.” I struggled to understand precisely what this statement meant. What are examples of processes that are more and less informative? \n\nI am confused about the connections to the heat equation, which is likely due to my own lack of understanding but may also indicate that the clarity could be better. My understanding is that we need to define a process that perturbs our data, $p(y | x)$. Such processes have been described in the ML literature, which the authors cite and can be solved through the matrix exponential. While normally this matrix exponentially may be hard to solve, since the noise process is applied independently across the dimensions, it should scale with O(S^3). It was unclear why small timesteps were introduced and why Euler integration was needed. Was the point that for some problems S^3 is too big and so for these problems we will restrict ourselves to small timesteps in order to avoid computing the matrix exponential? Overall, I was left confused about why we need to talk about heat equations at all and why we don’t just describe this as a CTMC with initial conditions? Is there something that the heat equation view is really buying us? \n\nLines 176-181 describe the subsection “Localization to random grid”. Related to my comment above about the lack of clarity regarding when the negative samples are “informative for training” the authors say that adding uniform noise to categorical variables “introduce too much noise to inform the EBM about the correlations in the data set.” Can the authors make this statement more precise in the text? I think I can intuitively see that if make random uniform perturbations at each dimension then you are sampling from a uniform distribution and this will be uninformative in some sense. However, I think this notion needs to be explicitly connected to the optimization objective / loss functional in order to make this clear. Furthermore, it is not clear why this isn’t solved by taking smaller timesteps so that only a few dimensions will on average be changed. Can the authors please clarify this?\n\nOverall, I am confused about the choice of time parameter and I think this needs to be better written in the manuscript. Section 3 establishes that as $t$ goes to infinity, “the energy discrepancy loss converges to maximum likelihood estimation.” The authors describe why maximum likelihood may have statistical advantages for finite data sets but then immediately move on in Section 3.1 to small timescales. This seemed like an odd transition and immediately made me ask “why not just use large times since this is maximum likelihood?”  I suspect the answer lays in Section 4.2 where it becomes apparent that the contrastive potential must be estimated with Monte-Carlo sampling. It seems that larger timescales induce higher entropy distributions that would require more MC samples to approximate the expectation on line 192? \n\nFor the related works, I felt the “Contrastive loss functions” paragraph needs more discussion. Energy discrepancies seems very closely related, if not exactly, to contrastive loss functions for training energy-based models. Can the authors please provide a more thorough comparison of these different methods?\n\nSimilarly, I did not see a discussion on pseudolikehood loss functions. For small timesteps, the loss function seems very closely related to pseudolikelihood estimation and it seems that when MC sampling must be used in this method to approximate an otherwise intractable integral, that the MC sampling can be seen as an MC approximation to pseudolikelihood? \n\nThe authors make a point of saying that ED is much more efficient that CD and point to timing experiments in the appendix. However, it appears that authors are only reporting timing for M=4 samples when in the paper M=32 is used. If I extrapolate and multiply the ED time by 8 (since 32 = 8*4) then ED is more expensive then all of these methods. Can the authors please clarify this? I suggest changing Table 6 to M=32 if that is what is used in the paper. \n\nThe biggest experimental win seems to come from the Tabular Dataset. I am not very familiar with this area so I have a limited ability to evaluate the significance here. While the results seem reasonable I have two questions: 1) since the baseline methods were published in 2019 are there more sensible baselines to compare with? I again emphasize that I am not requiring that the authors’ method is SOTA – it is okay if other methods beat their method. 2) Since these methods have mixed continuous and discrete data can the authors do a separate benchmark that only models the discrete columns? I think it would be helpful to tease apart whether the best strength of this method is in modeling mixed continuous-discrete data or also purely discrete data.  \n\nI was confused by the statement that method is sensitive to the assumption that the data distribution is positive on the whole space. Why is this more of an issue for ED than other EBM training techniques? Intuitively, it seems that you can always avoid the assumption that the data distribution is positive by just assuming that the energy in these regions is so high that the probability that you sample these regions is vanishingly low. Either way, can the authors point me to where this assumption of a low-dimensional manifold is investigated in the paper / SI?", "limitations": "I would like more honest discussion throughout the paper of the relative strength and drawbacks of their method. For example, while it is true that this method is “MCMC-free” it still requires Monte-Carlo estimation and the relative tradeoffs here are not adequately discussed. I would also like to see more comparisons of CD with large numbers of MCMC steps v.s. ED with large numbers of samples.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality: Energy discrepancy is a relatively new approach. While the original paper proposed some extensions to discrete data, this paper goes into extending energy discrepancy to discrete data in much more depth and includes new mathematical and experimental analyses. \n\nClarity: Overall, the paper is well written.  \n\nQuality: I believe the paper is technically sound. \n\nSignificance: The authors show on toy examples that their method outcompetes contrastive divergence. The authors appear to generally outperform two methods proposed in 2019 along with contrastive divergence. While I have some minor concerns about these baselines, outperforming these baselines is at least demonstrating some empirical benefit of this approach.", "weaknesses": "Clarity: The clarity can be improved a bit (see my questions below). \n\nSignificance: Despite demonstrating that the method can work empirically, I have some concerns with the overall significance. It seems that while the method works well on toy examples, the results are less impressive on real-world image modeling tasks. I am unfamiliar with the field of tabular data modeling and therefore, cannot properly assess the significance of the results. Beyond contrastive divergence the main baselines is a method from 2019 with 2,000 citations. Are there better baselines to compare against among these 2,000 citations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your questions! Due to lack of space we focus on questions of most interest for all reviewers. Citations and responses to the rest can be found in the comment.\n\n> Significance: [...] Are there better baselines to compare against [...]?\n> \n\nEBMs are appealing because they output the unnormalised data density explicitly, and allow more flexible modelling than e.g. normalising flows. This unlocks a unique set of downstream applications, see [1, 2, 3].\n\nGiven our interest in using EBMs as a tool in downstream tasks, we are not trying to claim SOTA compared to all generative models. For binary discrete image data, our method is competitive with other EBM training approaches like [4]. For tabular data we compare with [5] due to lack of EBM baselines. Prompted by your comment we found that a tabular DDPM gives a stronger baseline [6] and we will add this reference to our table.\n\n**Questions**\n\n> [...] What are examples of processes [q] that are more and less informative?\n> \n\nIn principle, the negative samples can be drawn from an arbitrary noise distribution. If, for example, $q$ is independent of the data, the contrastive term becomes just the log-normalisation constant up to a constant, i.e.\n\n$$\n\\log \\sum_{ x_-\\in  X} q( y) \\exp(-U_\\theta( x_-)) = \\log q( y) + \\log \\sum_{ x_-\\in  X} \\exp(-U_\\theta( x_-))\n$$\n\nwhich is an example of a **non-informative** $q$ distribution. In this case, ED becomes the log likelihood in theory, but the normalisation constant $Z_\\theta$ needs to be approximated from thousands of samples to be low-variance.\n\nAn informative noising distribution $q$ achieves two things.\n\n- $\\sum_{ x_-\\in X} q( y\\vert x_-)\\exp(-U_\\theta( x_-))$ can be computed from a small number of samples with lower variance than the full normalisation $Z_\\theta$ because the integrand is localised around $ y \\sim q( y \\vert x_+)$.\n- similar to GAN training, the model converges prematurely if the negative samples $x_-$ are trivial to distinguish from data $x_+$.\n\n> I am confused about the connections to the heat equation [...] It was unclear why small timesteps were introduced and why Euler integration was needed.\n> \n\nWe don't use the Euler discretisation in practice and our method is simulation-free, i.e. for any time $t$ we can sample exactly from the transition density $q_t(b|a)$ using the closed form solution in Proposition 1. For the uniform perturbation, the exact solution is equivalent to equation (7) via the time rescaling $t' := (1/S(1-e^{-St}))\\in [0, 1/S)$, (see comments). We are going to replace equation (7) with the closed form solution.\n\n> [...] Is there something that the heat equation view is really buying us?\n[...] “why not just use large times since this is maximum likelihood?”\n> \n\nAny CTMC could be used to define a noising process, but this leaves us with many choices for the rate function. In previous work using CTMC [7, 8], the structure on the discrete space is ignored and only the uniform and absorbing diffusion is discussed.\n\nWe choose the rate function as the graph Laplacian which defines the **smallest possible transitions** in the graph. For small $t$, the corrupted data points are then with highest probability the **nearest neighbours** of the data point in the graph.\n\nHowever, a small corruption may miss global distribution properties like mixture weights [9]. Theorem 1 tells us that for large $t$, ED performs MLE which captures these global properties at the cost of higher variance of the estimated loss. The heat equation allows us to trade-off the variance and better statistical properties of the loss.\n\nTo support our intuition empirically, we include new experimental results in the attached PDF, Figure 1, showing that a large $t$ requires increasing $M$ to achieve accurate energy estimates.\n\n> The authors make a point of saying that ED is much more efficient that CD and point to timing experiments in the appendix. However, it appears that authors are only reporting timing for M=4 samples when in the paper M=32 is used. [...]\n> \n\n$M=4$ is often sufficient to obtain good results, see table 8. For the main text of this paper we used $M=32$ to maximise our GPU usage. In fact, ED can be computed in parallel, while CD requires sequential sampling algorithms that cannot be parallelised. We provide the training time for $M=32$ below:\n\n|  | CD-1 | CD-5 | CD-10 | ED-Bern (M=32) | ED-Grid (M=32) |\n| --- | --- | --- | --- | --- | --- |\n| Per Epoch (s) | 29.1660 | 95.2178 | 167.5718 | 46.4317 | 44.0621 |\n\nIt shows that ED with $M=32$ is still more performant than CD-5. We will change the corresponding table in the revision.\n\n> [...] 1) since the baseline methods were published in 2019 are there more sensible baselines to compare with? [...] 2)\n> \n\nAs mentioned earlier, there is a better baseline that uses a diffusion model [6] which outperforms our approach on the tabular data benchmark. We provide the experimental results for detailed comparison:\n\n|  | Adult ↑ | Bank ↑ | Cardio ↑ | Churn ↑ | Mushroom ↑ | Beijing ↓ |\n| --- | --- | --- | --- | --- | --- | --- |\n| CTGAN | .861 | .774 | .787 | .792 | .781 | 1.01 |\n| TabDDPM | .910 | .924 | .802 | .793 | 1.0 | .570 |\n| TabED (Ours) | .853 | .845 | .796 | .814 | .985 | .978 |\n\nTabED refers to the best results from our proposed methods. We remark that, while the DDPM outperforms the GAN and EBM, we are interested in tasks that cannot be addressed by diffusion models.\n\n> I would like more honest discussion throughout the paper of the relative strength and drawbacks of their method. [...]\n> \n\nWe will revise the paper keeping your comment in mind. In brevity, the biggest strength of ED are improved theoretical guarantees over CD or robust behaviour when tuning an MCMC sampler is difficult (e.g. tabular data). The biggest drawback of ED is currently its scalability to data more high-dimensional than MNIST.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SMjnAs4CZ", "reviewer": "Reviewer_t2RA", "summary": "This article, \"Energy-based modelling for discrete and mixed data via heat equations on structured spaces\", proposes to perform the training on EBM, using the Energy Discrepancy (ED) loss, in the case where having multi-modal dataset mixing eventually continuous inputs but also discrete (categorical) ones. The work describes into details how to parametrize in different setting the inclusion of discrete variables, and they apply it to various datasets.\nThe main contributions are the design of the continuous time Markov chain transition probability that lies at the heart of the ED approach and the application to tabular dataset for which generative approach is usually hard.", "questions": "The authors said that their work is one of the first dealing with tabular data, I at least find this one dealing with EBM and tabular dataset: https://arxiv.org/abs/1912.09382, the authors might check if it is relevant w.r.t. their work. Also, this article https://arxiv.org/abs/2405.15376 deals with generative and categorical variables for EBM.", "limitations": "The limitations are correctly discussed in the article.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors show how their method can be efficiently used on Tabular dataset. In particular, they apply to several dataset and show that in average the EBM trained with Energy-Discrepancy using a discrete implementation of the Markov chain transition probability outperform the concurrent approach, ref[Xu et al 2019]. \n\nThe authors also show experimental results on image modelling.", "weaknesses": "The authors extend the formalism of Energy Discrepancy to the case of including discrete state in addition to continuous features. Whether or not this justifies an entire publication can be debated, Although it should be emphasized that the datasets under consideration are quite original.\n\nIt might be because I'm not an expert on ED, but while traditional EBM relies on MCMC to compute the gradient, ED does not. However, it is not clear to me if sampling the EBM trained in such way need MCMC to be generative ? If so, the article should provide more details on the implementation. They should also check that the trained model is at equilibrium (the generated samples corresponds to the equilibrium properties of the model).\n\nMore importantly, the comparison for the tabular dataset is only done at the level of the AUC curves. Can at least the authors compare the frequency and correlations amongst the generated samples and the true ones ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We first want to comment on the weaknesses of our paper mentioned in your review:\n\n> The authors extend the formalism of Energy Discrepancy to the case of including discrete state in addition to continuous features. Whether or not this justifies an entire publication can be debated\n> \n\nHere is why we hope that this work is interesting to the community: Currently, discrete EBM training methods are hard to tune and don’t translate to mixed state spaces. We want to introduce a simple performant training method to address these limitations.\n\nTo do so, we introduce a graphically informed noising process which may also be of great interest in other domains, e.g. discrete diffusion modelling, too. The theoretical understanding of the diffusion allows automated tuning of the time parameter $t$ as a function of the state space size (Theorem 2) which is a critical new contribution to this work substantially different from prior work and important to tabular data, where state space sizes typically vary across between features.\n\nOur theoretical analysis is supported by the experimental results that show good performance, thus unlocking a new tool for density estimation on non-continuous domains.\n\n> [...] it is not clear to me if sampling the EBM trained in such way need MCMC to be generative ?\n> \n\nTo use EBM as a generative model, one needs to solve two challenges: Training the model and sampling from the model. This paper addresses the training part, only. As you say, sampling methods have to be used to obtain synthetic samples from the model. We employ a fairly standard method for sampling with interleaved updates of the numerical and discrete state. The implementation can be found in the appendix (Algorithm 2).\n\nImportantly, tuning a sampler at inference time is much easier than tuning it for a stable training algorithm. Indeed, we were unable to implement the MCMC-based training for tabular datasets, while sampling from our model with MCMC was unproblematic.\n\nIt should be noted that EBMs are particularly useful in non-generative downstream tasks like out of distribution detection and calibration of classifiers, because unlike other generative models they output the unnormalised data density, explicitly. For this reason we focussed on contributing to better EBM training.\n\n> They should also check that the trained model is at equilibrium (the generated samples corresponds to the equilibrium properties of the model).\n> \n\nAs stated in lines 78-81, ED offers the theoretical guarantee that  $p_{\\theta^\\ast}(\\mathbf x) = p_\\mathrm{data}(\\mathbf x)$ after training converged. Hence, if we run an long-run MCMC sampler on our learned model with Metropolis Hastings acceptance step, the produced samples are at equilibrium by construction (The MH acceptance guarantees detailed balance which is equivalent to samples being at equilibrium.)\n\nThe same guarantees does not exist for MCMC-based training approaches because short-run MCMC has to be used in practice introducing biases in the optimisation, see e.g. [1]. For empirical evidence of these favourable properties, see figure 2, 7, 8 which contrast ED and CD results.\n\n> More importantly, the comparison for the tabular dataset is only done at the level of the AUC curves. Can at least the authors compare the frequency and correlations amongst the generated samples and the true ones ?\n> \n\nThanks for your suggestion. We include two new metrics here: single-column density similarity and pair-wise correlation similarity. These two metrics can measure the similarity of the frequency of single columns and correlations of pair-wise columns between the generated and the true table. They can be computed by using the open-source API: [SDMetrics](https://docs.sdv.dev/sdmetrics/reports/quality-report/single-table-api). It shows that the proposed ED approaches can outperform baselines or achieve comparable performance in most datasets.\n\n**single column  density similarity score**\n\n|  | Adult ↑ | Bank ↑ | Cardio ↑ | Churn ↑ | Mushroom ↑ | Beijing ↑ |\n| --- | --- | --- | --- | --- | --- | --- |\n| CTGAN | .814 | .866 | .906 | .901 | .951 | .799 |\n| TVAE | .783 | .824 | .892 | .899 | .965 | .711 |\n| TabCD | .719 | .790 | .824 | .845 | .618 | .799 |\n| TabED-Un | .791 | .760 | .948 | .905 | .918 | .843 |\n| TabED-Grid | .751 | .766 | .945 | .846 | .951 | .951 |\n| TabED-Cyc | .778 | .826 | .937 | .834 | .969 | .751 |\n| TabED-Ord | .828 | .894 | .933 | .887 | .943 | .747 |\n\n**pair-wise correlation similarity score**\n\n|  | Adult ↑ | Bank ↑ | Cardio ↑ | Churn ↑ | Mushroom ↑ | Beijing ↑ |\n| --- | --- | --- | --- | --- | --- | --- |\n| CTGAN | .744 | .769 | .717 | .826 | .828 | .761 |\n| TVAE | .669 | .772 | .687 | .808 | .919 | .618 |\n| TabCD | .522 | .600 | .629 | .710 | .428 | .761 |\n| TabED-Un | .653 | .661 | .828 | .851 | .825 | .735 |\n| TabED-Grid | .583 | .768 | .829 | .764 | .842 | .842 |\n| TabED-Cyc | .636 | .703 | .810 | .755 | .860 | .685 |\n| TabED-Ord | .702 | .796 | .811 | .791 | .826 | .662 |\n\n> The authors said that their work is one of the first dealing with tabular data, I at least find this one dealing with EBM and tabular dataset: [...] Also, this article https://arxiv.org/abs/2405.15376 deals with generative and categorical variables for EBM.\n> \n\nThank you for these interesting references! We missed the first reference in our literature research. Unfortunately, we are using different benchmark data sets so a direct comparison is difficult. Furthermore, RBM architectures tend to be more constrained than deep NN architectures, which has the advantage of stabilised training at the cost of flexibility. This means that ED and the referenced work have slightly different advantages and drawbacks.\n\nThe second article is very interesting. It is concurrent work to ours and was published after the NeurIPS deadline, so we were not aware of it.\n\n[1] Nijkamp et al. Learning non-convergent non-persistent short-run\nMCMC toward energy-based model", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments. First, we would like to summarise the strengths of the paper according to the reviewers.

The reviewers agree that our paper is a successful extension of energy discrepancy to discrete and mixed data where energy-based modelling is challenging.

> **9ZAB**: The paper successfully extends the Energy Discrepancy method to discrete and mixed data with solid theoretical analysis, addressing the challenge of robust and fast sampling in these space.
**qwnm**: The paper proposes a relevant extension to recently published work. Especially Theorem 1 does not seem obvious, and the paper may open up the use of the Energy Discrepancy loss to a much wider variety of use-cases.
**FxWc**: While the original paper proposed some extensions to discrete data, this paper goes into extending energy discrepancy to discrete data in much more depth and includes new mathematical and experimental analyses.
> 

The reviewers generally consider our paper to be well-written and well-supported by experiments.

> **9ZAB**: The designed experiments demonstrate the method's ability to accurately capture complex data structures and generate high-quality synthetic data, highlighting its practical applicability.
**qwnm**: The results are good compared to standard contrastive divergence based methods. The paper is well written, and I found it easy enough to understand even without prior knowledge on the Energy Discrepancy method.
**FxWc**: Overall, the paper is well written. The authors appear to generally outperform two methods proposed in 2019 along with contrastive divergence. While I have some minor concerns about these baselines, outperforming these baselines is at least demonstrating some empirical benefit of this approach.
**t2RA**: The authors show how their method can be efficiently used on Tabular dataset. In particular, they apply to several dataset and show that in average the EBM trained with Energy-Discrepancy [...] outperform the concurrent approach.
> 

We now want to address the biggest concerns and questions of the reviewers.

- The reviewers **9ZAB, qwnm, FxWc,** and **t2RA** were concerned about the scalability of our proposed method, particularly for image data. To address this concern we want to put our work into perspective of the research field.

Scalability is a valid concern for EBMs in general. Even in continuous spaces, EBMs struggle to scale up and outperform the sample quality of GANs and diffusion models. However, EBMs are appealing because they explicitly output an approximate data density, which is advantageous for downstream tasks such as out-of-distribution detection and calibration. 

Most existing EBM training methods rely on contrastive divergence which over-smoothes the estimated densities. Our method is more reliable due to theoretical guarantees, requires little tuning, and displays competitive performance on binary image data. ED can be computed in a single parallel pass, unlike MCMC-based methods that need many sequential steps. Thus, ED is significantly cheaper and more parallelisable. Furthermore, MCMC-based approaches have yet to be extended to mixed-state spaces, and many tabular data sets of interest have a comparably small number of features.

Enhancing the scalability of EBMs will likely require a combination of EBMs with other tools such as a variational decoder model used to scale energy discrepancy in the continuous domain in prior work. The energy function could also be learned in a diffusion model style with the energy discrepancy loss at various noise scales, thus improving training stability and learning a sampler alongside the EBM. However, this was not the main concern of our work.

- The reviewers **9ZAB, qwnm,** and **FxWc** had questions about the importance of the noising process which we define as a heat equation on a graph. We want to summarise what guides the choices we are making.

In principle, the negative samples can be drawn from an arbitrary noise distribution. If, for example, $q$ is independent of the data, the contrastive term becomes just the log-normalisation constant up to a constant, i.e.

$$
\log \sum_{ x_-\in  X} q( y) \exp(-U_\theta( x_-)) = \log q( y) + \log \sum_{ x_-\in  X} \exp(-U_\theta( x_-))
$$

which is an example of a **non-informative** $q$ distribution. In this case, ED becomes the log likelihood loss in theory, but the normalisation constant $Z_\theta$ needs to be approximated from thousands of samples to be low-variance.

An informative noising distribution $q$ achieves two things.

- $\sum_{ x_-\in X} q( y\vert x_-)\exp(-U_\theta( x_-))$ can be computed from a small number of samples with lower variance than the full normalisation $Z_\theta$ because the integrand is localised around $ y \sim q( y \vert x_+)$.
- similar to GAN training, the model converges prematurely if the negative samples $x_-$ are trivial to distinguish from data $x_+$.

If we select $q$ via an arbitrary CTMC we have no guidance about what rate function to choose. In previous work using CTMC the structure of the discrete space is ignored and only the uniform and absorbing chain is discussed. We choose the rate function as the graph Laplacian which defines the smallest possible transitions in the graph. For small $t$, the corrupted data points are then with highest probability the nearest neighbours of the data point in the graph. This guarantees that negative samples are different from data while staying as informative as possible.

However, a small corruption may miss global distribution properties like mixture weights. Theorem 1 states that for large $t$, the difference between the optimisation landscape of ED and MLE goes to zero. MLE has preferable statistical properties, but taking the large $t$ limit comes at the cost of higher variance of the estimated loss. Therefore, the heat equation provides a unified framework and allows us to tune the loss properties in terms of a single hyperparameter $t$.

Meta-review: Energy-discrepancy is a recently loss function similar in spirit to contrastive divergence, but with a more clear loss (as opposed to algorithmic) formulation and more tractable for analysis. The paper extends the energy-based discrepancy to models involving discrete states, with some convergence analysis. (The analysis is intuitive—e.g. if the mixing rate is faster, the method provides a better approximation.) There is also some analysis for particular state spaces. Reviewers appreciated the new method, but had some concerns about the experiments (in particular scalability).

Small point: Regarding theorem 1, the authors might wish to stress in the paper that z_t is independent of θ, meaning that when the right hand side is small, the two losses are nearly equivalent regardless of the value of z_t.
Decision: Accept (poster)