Incorporating Inductive Biases to Energy-based Generative Models
Authors: Yukun Li
Liping Liu
Venue: Accepted by TMLR ()

Abstract: With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.

Published in Transactions on Machine Learning Research (10/2024)
Incorporating Inductive Biases to Energy-based Generative Models
Yukun Li
wikumlli@tufts.el
Deportment of Computer Science Tufts University
Li-Ping Liu
liping. live Suufts edu
Department Computer Science Tufts University
Reviewed on OpenReview: https: opwp:evieeee........mmm net/ net/forum? forum? id= k982Db
Abstract
With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modcling. Specifically, we augment the energy term with parameter-free statistic function to help the model capture key data statistics. Like exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approxinately max- imizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Fur- thermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.
1 Introduction
Easegy-based models (EBMs) MMrrpy 2012 LeCun et al., 2006; Ngiam et al. 2011 which provide a
flcxible way of parameterization, are widely used in data modeling such as sensitive cstimation (Nguyen
& Reiter 2015 Wenliang et al., 2019 Jiang & Xiao 2021), structured prediction (Belanger & McCallumi
2016 He & Jiang 2020 Pan et al. 2020). and anomaly detection (Zhai et al. 2016 Liu et al.] 20200) They
recently have achieved successes as generative models for data of different modalities, such as images Du &
Mordatch 2019 Vahdat & Kautz
2020), texts (Deng et al. 2020 Yu et al.
2022), graph (Liu et al.
2021
Chen et al. 2022 Xu et al. 2022 and point cloud (Xie et al. 2020 2021 Luo & Hu 2021 thanks
to the
advent of new training methods of score matching (Song & Ermon 2019 Song et al. 2020b Ho et al.
2020
Previous EBMs often have linear energy functions in model parameters, so they fall into the category of exponential-family models (Wainwright et al., (208).. An exponential family model has a few nice properties. It matches data statics when it maximizes the likelihood of the training data. Among all models, also has the largest entropy with the constraint of matching statistics. When constructing an EBM, statistic functions are often used to capture key statistics of the data. However, in recent years, EBMs have used neural networks as their energy functions. Since their energy functions are not linear in trainable parameters, they are not exponential-family models anymore.
In our construction of EBMs, we propose to include linear terms defined with special statistic functions the energy function retrieve some good properties of exponential-family models. Even with complex data types such as point clouds and graphs, we still often have domain knowledge specify statistic functions that the model can capture desired properties. Therefore, the linear term with special statistic functions

---

Published in Transactions on Machine Learning Research (10/2024)
allows us to inject inductive biases in model fitting. This framework applics to the domains where prior information can be written in statistic functions.
The proposed method is very useful for data fitting in multiple iterations. Note that data fitting is often an iterative procedure (Gelman et al.) 1995), and the goodness of fit is often monitored by the discrepancy between statistics computed respectively from model samples and real data. With our method, we can improve the model by adding corresponding terms to narrow the discrepancy. Taking molecule generation an example, when we observe molecule samples violate the valency constraint, we can incorporate the knowledge of valency rules to improve the model.
We evaluate the effectiveness of this new approach on three tasks: modeling molecular data, fitting handwrit- ten digits, and modeling point clouds. The results demonstrate the effectiveness of the proposed approach in a wide range of pata-fitting domains.
2 Related Work
The need incorporate inductive bias into generative model has drawn researchers' attention for a long time. Zhu et al. (1997) proposes the general principle for incorporating prior knowledge into generative model. KKalifa et al. (2020) introduces novel method for text generation using EBMs to enforce desired statistical constraints. Their framework employs the EBM model to approximate pre-trained language model instead of directly fitting the data. Korbak et al. (2021) extends this approach to code generation, imposing a constraint that generated sequences are compilable. Qin el. al. (2022) applies energy constraints to a transformer decoder to control text semantics and style. Lafon et al. (0023 learns the hybrid model combined with the Gaussian Mixture Model (GMM) and EBMs for ont-of-distribution detection. In contrast, our work focuses on improving data fitting with inductive bias.
Another thread of research changes generative model's behavior during the sampling stage Dhariwal & Nichol 2021 For example, Chung et al. (2022) proposes to manifold constraint to guide pre-trained
diffusion model's sampling path. Kim et al. 2022 Zhao et al. (022] ; Song et al. [2022] Yu et al. 2023 Liu
ett al. (2023) Bansal et al. (2023 Li & Liu
[2024] propase different guidance for different specific problems
to boost the performance of each task. In contrast, our method aims to the original data distribution more
accurately by adding the statistics term. Notably, the classifier guillance method operates during inference
with manually tuned weights, whereas our model learns the inductive bias strength corresponding to data
distribution statistics.
3 Background
3.1 Score-matching for energy-based models
An energy-based model is defined with an energy function E, parameterized by 0,
px(x)= exp(- exp(-Fo(xx)
(1)
Zo
Here, x E Rd, is the feature dimension. Ze is the partition function: Ze = exx((=E which is usually intractable. Various methods exist train an energy-based model. Maximum likelihood training with Markov Chain Monte Carlo (MCMC) sampling (Hinton) 2002) is one standard method to train the baseg model, but MCMC is typically computationally expensive.
A modern EBM often devises the energy function (x with a neural network, and Ô represents learnable parameters of the neural network (Song & Kingma 2021). These models are usually fit by score matching (Hväärinen & Dayan 2005 Kingma & Cun 2010), which avoids the explicit calculation of the Zo. Score
matching minimizes the mean square error belween the model score function So Vx logpooxx and the
data score function Vx log Paota(
(Paata (x) (x) (llo = Expaaa (x)
Vx log Paiaa (x) Vx log
(2)
2
2

---

Published in Transactions on Machine Learning Research (10/2024)
where (x) (x) is the true data density function, pe(x) is the model density function. To bypass the in- tractable of the first term, we often use integration by parts to rewrite it as
1 2
d
[E((x)
0 Ex(xx) (dz²)²
Dy (pdata ( (x)( = Epacea (x)
+ const.
Ox
Several efficient variants of the score matching have been proposed. One Sliced Score Matching (Song et al., 2020a), which randomly samples a projection vector v, takes the inner product between v and the two scores, and then compares the resulting two scalars:
Dr(paata (x)laaaaxx)PPooX))) - (xp Ep(y)
(v Vx log Pdata (x)- - v Vx log (x))3
(4)
The parallclizable variant of sliced Score Matching is Finite difference sliced score matching (FDSSM) (Pang ett al.] 2020). Another score-matching variant Denoising Score Matching (Vincent) 2 (DSM). DSM
avoids the computation of the second-order derivatives by adding some small noise perturbation to the data. Then DSM learns the noisy data distributions g(x) x Pdota (x):
Dr(g( (p( = Ey(x.8)
V log (x|x) - Vx po (x)1
(5)
2
3.2 Exponential-famity distributions
A distribution from the exponential family can be viewed as special type is linear in ils parameters. Various common distributions such as Gaussian, Bernoulli, and Poisson are in Exponential-family distributions. Here we consider continuous distributions whose probability density fanction (PDF) takes the following form:
type of EBM whose energy function
p(x:n)= = exp(1) T(x))
(6)
Z(n)
In this form, we assume the base measure is 1. The statistic function T(x) decides the distribution type, while the parameter 7 decides the actual distribution of that type. The partition function Z(n), which is often hard to compute, ensures that the PDF integrates to over the domain of the distribution.
The statistic function T(x) is also meaningful in capturing data statisties. The statistics 1 E T(x;) computed from a dataset (x;: 1,.,,, is called the data's sufficient statistics, which provide sufficient information about the distribution parameter. Data fitting with p(x; n) is centering at sufficient statistios: if the data likelihood maximized with a particular n under the model p(x; n), then Ep(xm [T()) = N DN =1 T(x:).
4 A Hybrid Energy-based Model with Inductive Bias
This work proposes combining neural energy functions and interpretable statistic The hybrid model takes the following form:
statiiss functions to develop EBMs.
(x) exp (Fo(x) n Tx())
Z(0,n) n)
Here Z(0, = fg exp (Fo(x) + 17 Tx()) dx is the partition function. Fx(x) E R is a real-valued function constructed with neural network parameterized by 0. The statistic function T(x) itself has no learnable parameters, but it is multiplied to the learnable parameter v in the linear term.
The hybrid model still possesses nice property of the exponential family distribution. Model fitting still aims to match the distribution mean of sufficient statistics to the sample mean (Wainwright et al.) 2008 Let 1(0, v)) denote the log- likeeiio of the data, A(O,n) = log (0,n):
N
1(0, n) = Σ Fo(x,) +
Σ T T(x;) - NA(0,))
(8)
1=1
3

---

Published in Transactions on Machine Learning Research (10/2024)
Theorem 1. If the parameter n is at local marimum of the I(0,n) n) for fized 0, then Epo.n [T(x)] N EE T(x.).
Proof. If we treat exp (Fo(x)) as a base measure, then po,n (x) is an apponential-failly distribution. Then, the gradient of data likelihood with respect r can be derived according to Wainwright et al. (2008) (eq. 3.38 and Prop. 3.1). By taking the derivative of L(0, 77) with respect n7, we have
V Σ T(x:) N Eppen]
(9)
If the weight is at the local minimum, then V which leads to the conclusion.
With this property, will construct a hybrid model with specific stat in its linear energy terms. Model fitting will pay attention to the specified statistics in the data. It can be viewed approach to injecting inductive bias into the model. Before we discuss such models in real applications, we first consider their training procedure.
4.1 Training the model
We choose the score matching method to train the EBMs given its efficiency and effectiveness Song et al. 2020b). The score function for this hybrid model is:
S0. x) +xFo(x)+ Vx((x) "
(10)
We can either take derivative Fx(x) to get (x), or directly specify (x) without specifying F (x).. In the latter case, Vx)o(x) is approximated by neural network with parameter 0. Once we are here, we follow the standard procedure of denoising score matching (Vincent) 2011) to train our model:
Ex-p(x) Bxwag (x
Vx log go (x(x) - se.n (x)
(11)
Where & is the perturbed data. To learn the function better in the low-density regions, adopt the technique by perturbing the data with K different noise levels {o learn the noise-condittoned score network (ong & Ermon 2019) with the following loss:
K
[(0.n:
Σ X(o,) (x) /l log 9., (x(x) sa, (x)
(12)
K
Here X(o;) is the weight for each noise level.
Given the approximation here, one question whether the trained model still matches the data statistics. Note that the matchng loss the lower bound of the likeliiel (Song et al. 2021 with appropriate weighting X.
on L(0, A)
(13)
Model training with score matching approximately maximizes the data likelihood. With a similar spirit, the learned model approximately matches the data statistics. Furthermore, the difference between the distribution mean and the data mean: AT(x) Ex-ppen. [Tx] Ex-ppoos [Tx] approaches to zero the model parameter n approaches local maximum of the log- likelihood. This can be proved based the convexity of the log-likelihood function w.r.t. parameter as a base measure, then the hybrid distribution in (7) falls in the exponential family of distributions, and its log-likellnn likelihood convex w.r.t. 77 by Wainwright et al. (2008). Therefore, parameters (0*, (0",7") at local maximum of the likelihood indicates that 17" at the maximum of the likelihood function with 0 fixed. From convex analysis (Boyd & Vandenberghe 2004), the norm of the gradient approaches zero as (0, approaches the local maximum.
For any 0 fixed, exp (Fo(x) can be viewed

---

Published in Transactions on Machine Learning Research (10/2024)
4.2 Inject inductive bias through the function T(x)
As in an exponential-amily model, the statistic function points the hybrid model's attention to special statistics of data distribution. Therefore, it is a convenient approach to inject inductive bias into the model. Without any restrictions over the function form, the statistic function is a convenient approach to express such inductive bias. In this section, we leverage this property to develop models for three applications to show its practical value.
4.2.1 Statistic function for fitting molecule data
A molecular generative model is an important tool for chemical applications, and it is often defined as an EBM (Liu al. 2021 Niu et al., 2020 Jo et al., 2022). Here, we consider two molecular generative models, EDP-GNN Niu et al. (2020) and GDSS Jo et al. (022), which both can be viewed as energy- baagyybasedd models and specify distributions of molecules through their molecular graphs and atom types. Assume the atom has k lypes. In these two models, the random variable x (b, A) represents a molecule graph containing n nodes (atoms), with b E .k....,]'' representing atom types in a molecule, and A being the adjacency matrix representing the connectivity of corresponding atoms. Without the statistic function T, these two models use a neural network to define (x))
An energy-based model based on a neural function often assigns non-zero probabilities to "invalid" molecules even though the data contains zero such examples. In particular, all molecules in the data satisfy valency constraints. For example, a carbon atom has four bonds, an oxygen atom has two bonds, and a nitrogen atom has three or five bonds. However, samples from an EBM defined by neural energy function often violate the valency constraint. While there are methods employing other techniques to enforce valency constraints in the sampling stage (Zang & Wang] 2020), we consider such constraints in a canonical approach to data fitting. Specifically, we express the valency constraint as a statistic function: a valid molecule has zero value from the function, while an invalid molecule has positive value. Let the constant vector v E (1,... v/ store valences for l atom types, with v being the maximum valence possible. Then, the valeney constraint is:
A x 1 S on)y
Here onchot(b) represents cach node with a onc-hot vector that indicates its atom type, then onchot(b)) (b)v is a vector indicating valency values of all atoms in the molecule. From this constraint, we define the statistic function as:
T(x)= 1 x max(0,A A x onchot(b)v):
(14)
A valid molecule always has Txx) = O, while an invalid molecule violating the valency constraint has T(x) > 0.
As discussed above, if the model fits the data well, the expected value of T(x) should be O. Thus, the valency constraint is imposed over samples from the model. The neural energy function may also learn to generate molecules similar to the training data. Our new statistic function T(x) makes the preference explicit in model fitting.
6
3
2
4
8
4.2.2 Statistic function for fitting handwritten digits
In the handwritten digits dataset LeCun (1998), we observe that the mar- gin area surrounding the digit has pixels all taking value 0. Fig Sig[[shn shows some examples of handwritten digit images, and it can be observed that the pixels in the margin are all zeros. A model with this knowledge will better fit the data. We introduce a statistic function defined as the sum of pixels located at the boundaries of the image. Let x E Rac represent one image, where h and h are the height and width of the image, respectively.
2
8
6
Figure 1: All pixels outside the
yellow bounding box are zero. This piece of prior knowledge is encoded in the statistics in 15
Let be the width of the margin.
T(x)= Euses [x()
(15)
|S|
5

---

Published in Transactions on Machine Learning Research (10/2024)
Here S represents the set of pixels in the margin area S {(t,j)l - or ori2 h- j or 1 h With this statistic function, the model will explicitly consider pixel values in the margin area during the learning process.
4.2.3
Statistic function for fitting point clouds
Point clouds have been widely used in computer graphics, computer vision, and robotics (Achlioptas et al, 2018 Xiao et al. 2023 He et al.) 2023 Huang et al. 2024). Several algorithms have been proposed for point cloud generation. In this work, we consider the DPM (Luo & Hul 2021) and latent diffusion model. These Iwo methods can be taken as directly modeling the joint distribution of all the points, meaning that Fx) is the gradient of all the points' joint distribution in our framework.
For the statistic term, considering smooth surfaces are a characteristic trait of high-quality point clouds, we define the following statistic function for the model:
T(x) tr(x Lx).
(16)
Here x E R Nx3 is the 3D coordinates of N points. L is the sparse Laplacian matrix of the point clouds' k-nearest. neighbor (k-nn) graph. The k-nn graph is constructed using a kd-tree, with a time complexity of On log (Preparata & Shamos, 2012), where n represents the number of data points. This computa- tionally more efficient than the naive k-nn method and can scale better. The function T(x) computes the differences between node and its neighbors in the graph. It is measure of the uniformness of points in the point cloud. With this statistic function, the model fitting procedure will explicitly adjust the uniformness of generated point clouds to the same level training samples.
5 Experiments
We evaluate the effect of our special statistics data fitting through three generative tasks: molecule graph generation, image generation, and point cloud generation. Experiments related to molecule generation are detailed in Section 5.1 Section 5.2 discussed the image generation experiments. Section 5.4 discussed the point cloud generation task. Each section briefly discusses each lask's experiment setup and evaluation metrics we used. More details are in the Appendix.
5.1 Fitting molecular data
In the first experiment, we incorporate our statistic function derived from valency constraints into an EBM model for molecule generation.
Our model evaluated on the QM9 dataset (Pinheiro el al. 2020), , which comprises diverse collection of 133,885 molecules, each containing up maximum of 9 atoms. Following previous studies, the molocules removed. Two essential metrics employed to evaluate the quality of the generated molecules: validity and valency ratio. A valid molecule must satisfy certain chemical constraints and rules, indicating chemically reasonable structure. Validity is determined by calculating the proportion of valid molecules without valency correction or edge resampling. We evaluated our strategy on two methods: EDP-GNN (Niu et al.) 2020) and GDSS [Jo al.) 2022). We trained GDSS for 300 epochs, the batch size 1024, and the learning rate is 50-3. We trained EDP-GNN for 1000 epochs, the batch size is 1024, and the learning rate 8c-3.
Experiment Settings:
kekulizated using the RDKit library (Bento et al., 2020), and hydrogen atoms are
Results: We first evaluate the validity ratio of generated molecules. The results are shown in the left column of Tab. [ Ten thousand molecules are sampled for evaluation. The results show that the model improves the validity ratio of the samples with our new term expressing the validity information. Wo further check expectation E [T(x)) in two distributions with and without the statistic function. Let Ar()) = E Epmodet [Tx)]] - Ep.ras [x) denote the difference between the distribution mean and the sample mean of the statistics term. The results clcarly show that our model pays attention to the new term and tries match its mean to the sample mean of the data. It demonstrates that the extra valency term does help the model learn the valency property.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Results for molecule generation on Q.99 dataset
QM9
Validity ratio(%)1
(1)
EDP-GNN
88.33
1.85
EDP-GNNNwith T(x))
94.52
0.98
GDSS
95.72
0.94
GDSS(with T(x))
96.73
0.93
Table 2: Negative Likelihood performance on MNIST dataset
Method
NLL (J)
4T(x) (4)
VE-SDE
3.56
6.52
VE-SDE(will (with T(x))
3.49
6.42
VP-SDE
3.37
6.48
VP-SDE(with T(x))
3.29
6.37
5.2 Fitting data of hand-written digits
We evaluated the effectiveness of our method the MNIST dataset, widely used benchmark for vari- ous image downstream tasks. Each image in the MNIST dataset is 28x28 grayscale representation of a handwritten digit from 0 to 9.
Experiment Settings: We experimented on the two forward SDEs: VP-SDE and VE-SDE. The boarding pixels are extracted using a mask. The size of the mask is 22 x 20. The model is trained for 1000 epochs and then compared to the likelihood. The batch size is 4096, and the learning rate is 1e-2. The learning rate is kept constant for the first 300 epochs and decreases linearly from 300 1000 epochs. The parameter for the statistics term 7 is initialized to be zero. We assess the performances of competing models by comparing their likelihoods on the test. set.
Results: We evaluated our model via the test set negative bog-likel (NLL) in terms of bits/dim (bpd). Tab. 2 reports the averaged NLLs of probability flow ODE Song et al. (020b) over two repeated runs of likelihood computations. The results suggest incorporating the statistics term into the model improves the likelihood. This improvement implics that the model's distribution aligns better with the observed data when considering the inductive bias. The results also show that the difference between the distribution mean and the sample mean of the new statistic is smaller with our method, which implies that our model better captures the data statistic.
5.3 Fitting data of small grayscale images
We tested our approach on the FashionMNIST dataset Kayed al. (2020). Similar to the MNIST dataset, added the constraint the boarding pixels. Let R represent one image, where h and h are the height and width of the image, respectively. Let h and w represent the height. and width of images in the dataset, and a be the width of the margin, then we use the same statistic function T(x) in (15) that computes the gray level of the border area of an image. In real data, the border area contains mostly white pixels. Without any specification, a typical generative model often overlooks such patterns. Here, we explicitly emphasize this pattern in the model that fits with this statistic function.
Experiment Settings: We evaluated performance using the VE-SDE method with batch size of 64 and initial learning rate 1e-2. The learning rate decays step-wise by multiplying it by 0.95 every ten epochs. The model was trained for 100 epochs. Boarding pixels were extracted using a 21 x 19 mask. We repeated the experiments with different random seeds three times and averaged the results.
We assess the performance by comparing the negative likeliiiddood (NLL) on the test set and the deviation between the distribution mean and the sample mean of the statistics term (AT(z) against baseline methods. The results, presented in Table 3 indicate our method's performance improvements in NLL and the statistical term difference compared to the VE-SDE without T(x).
Results:

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Negative Likelihood performance on FashionMNNIST dataset
Method
NLL ())
(+)
VE-SDE
4.605
47.12
VE-SDE(with T(x))
4.599
43.75
Table 4: Comparison of shape generation on ShapeNet. MMD is multiplied by 10 COV is multiplied by 102. Mean diff is the difference between the sample mcan and the distribution mean
Category
Model
MMD (x10².
COV (x102.1)
1-NNA (%,,l)
AT(x) (L)
DPM
0.572
43.75
86.91
102.71
(wt(w T(x))
0.542
45.50
85.25
91.58
Airplane
Latent diffusion model
0.389
49.11
68.89
39.90
Latent diffusion model(with T(x))
0.387
49.60
67.04
37.52
DPM
1.140
34.94
79.97
326.65
DPM(with T(x))
1.137
36.83
75.19
284.98
Car
Latent diffusion model
0.802
43.70
76.23
203.02
Latent diffusion model(witee T(x))
0.781
45.31
73.3
187.90
5.4 Fitting data of point clouds
Point cloud datasets are typically collections individual 3D points, each with its position in space and potentially associated attributes, forming a sparse and irregular represontation of the underlying scene. Uni- formity is a critical factor for generating point clouds. Clumping often occurs when points are concentrated in specific areas, potentially losing detail in other regions. On the other hand, sparsity results in areas with few points, leading to information loss and inaccuracies. Therefore, generative model needs to pay attention to uniformness among data points to achieve premium results.
Experiment Settings: We evaluated our model on the ShapeNet dataset (Chang et al. 2015) a widely used benchmark in 3D shape analysis and understanding. The ShapeNet dataset comprises 51,127 unique objects distributed across 55 categories. Each category represents a distinct class of objects, covering various shapes, such as airplanes, cars, chairs, and animals. The dataset's richness and diversity make it ideal for evaluating generation performance. We evaluated our strategy on the two models. One is the DPM model latent diffusion model with the decoder is also the score-
[Luo & Hu 2021), and the other is conducted based model. The latent code is the shape code for each input point cloud. We use PointNet (ee et al. 2017) to map the input points into a 512-dimension latent feature code for the encoder model. For the decoder and the latent prior score model, we use OccNet (Mescheder et al. 2019), which stacked 6 ResNet blocks with 256 dimensions for every hidden layer. We evaluated our model in two categories: airplane and car. We report Minimum Matching Distance(MMD),, Coverage score(COV), and 1-NN classifier accuracy(I-NNA) evaluate the quality of the samples.
The results are shown in Tab. The results demonstrated that integrating smoothness constraints into generating and processing point cloud data improves generation quality. Encouraging a more even distribution of points makes the resulting point cloud representation more accurate, robust, and suitable for various applications. We further evaluate the difference between the sample mean and the distribution mean of the new statistic. We see that our method reduces the gap, which indicates that our model better captures key statistic in the data.
Results:
5.5
The quality of the statistic function
One question is whether an arbitrary function can act as a statistic term and improve the data fitting performance. The question is important because if an arbitrary statistic function can improve the model performance, is easy for neural model F(x) to pick up such statistics in the energy function. In this section, we set up two types of statistic functions: the first type captures data statistics meaningfully, and the second type has no obvious relationship with the data. For the first type, still the statistic
8

---

Published in Transactions on Machine Learning Research (10/2024)
0.06
0.05
0.02
0.00
0.00
500 800
opoch
Figure 2: The comparison between 7) values respectivcly for arbitrary statistic sin(1 and the mask statistic specified by 15 The result indicates that only specially designed statistics are likely to help the model learn.
function specified by [5 For the second type, we set T(x) = sin(1Tx] which doesn't 't seem to capture any reasonable data statistics. We then learn two models respectively with the two statistics and check their corresponding parameters 17. We conduct this experiment on the MNIST dataset.
We get the results in Fig 2 where we plot 1) values against training epochs. We see n is positive for the statistic specified in (15p. As comparison, y) corresponding the meaningless statistics sin(1 x) converges to zero, which means that it does not help the model to learn.
Another question arises: how to select meaningful statistical function for specific problem to help the model to learn? One approach is to leverage the domain knowledge associated with the problem. For instance, in the context of molccular graph generation, the chemical valency constraint is a critical factor to consider. The domain constraints can be formulated as the statistic function of the generative model to enforce such knowledge.
6 Conclusion
In this work, we propose an energy-based model whose energy function includes a neural energy function and a specially designed statistic function. We show that this hybrid model inherits the nice property of exponential-fanee
models, that is, model training approximately matches the statistic's distribution mean sample mcan. This property allows us express special statistics in the energy function as an inductive bias. We have shown that this technique be extensively applied to multiple applications. The experiments show that the proposed strategy improves the modeling performance three data types, molecular graphs, hand-written digits, and point clouds.
Acknowledgments
We sincerely thank all reviewers and the editor for their insightful comments. The work was supported by NSF Award 2239869.
References
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843 852, 2023.

---

Published in Transactions on Machine Learning Research (10/2024)
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning, pp- 083-992. PMLRR 2016.
A Patricia Bento, Anne Hersey, Eloy Félix, Greg Landrum, Anna Gaulton, Francis Atkinson, Louisa Bellis, Marleen De Veij, and Andrew R Leach. An open source chemical structure curation pipeline using rdkit. Journal of Cheminformatics, 12:1-16, 2020.
Stephen Boyd and Lieven Vandenberghe. Conver oplimization. Cambridge university press, 2004.
Angel X Chang, Thomas Funkhouser, Lconidas Guíbas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rict 3d model repository. arXiv preprint Xin: 1612:51..0000 5120030,, 2015.
Xiaohui Chen, Yukun Li, Aonan Zhang, and Li-ping Liu. Nvdiff: Graph generation through the diffusion of node vectors. arXiv preprint 1eXim:2211.0771 2022.
Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint orXiv:2206.009, 00941, 2022.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Mare' Aurelio Ranzato. Residual energy-based models for text generation. orXiv preprint 2rXiv:2044.11 11714, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Yilun Du and Tgor Mordatch. Implicit generation and modeling with energy based models. Advances in Newral Information Processing Systems, 32, 2019.
Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Boyesion dala onalysis. Chapman and Hall/CRC, 1995.
Wenchong He and Zhe Jiang. Semi-supervised learning with the em algorithm: A comparative study between unstructured and structured prediction. IEFE Transactions on Knowledge and Data Engineering, 34(6): 2912 2920, 2020.
Wenchong Hc, Zhe Jiang, Tingsong Xiao, Zelin Xu, Shigang Chen, Ronald Fick, Miles Medina, and Christine Angelini. hierarchical spatial transformer for massive point samples in continuous space. Advances in neural information processing systems, 36:33365-33375 2023.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation, 14(8):1711-1800, 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing syslems, 33:6840 6851, 2020.
Zhang.Jin Huang, Yuxin Wen, ZiHao Wang. Jinjuan Ren, and Kui Jia. Surface reconstruction from point clouds: A survey and benchmark. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-20, 2024. doi:
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Renyan Jiang and Shihai Xiao. Performance comparison of three parameter estimation methods heavily censored data. In 2021 8th International Conference on Dependable Systems and Their Applications (DSA), pp. 295 301. IEÉE, 2021.
Jachyoong Jo, Scul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, pp. 10362-10383 PMIRR 2022.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Mohammed Kayed, Ahmed Anter, and Hadeer Mohamed. Classification of garments from fashion mnist dataset using cmn lenet-5 architecture. In 2020 international conference on innovalive trends in commus nication and computer engineering (ITCE), pp. 238 243. IEEE, 2020.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text gencration. arXiv preprint Xrx:22012. 11635, 2020.
Gyeongnyeon Kim, Wooseok Jang, Gyuseong Lee, Susung Hong, Junyoung Seo, and Seungryong Kim. Dag: Depth-aware guidance with denoising diffusion probabilistic modcls. arXiv preprint (xXi:2212.08861, 2022.
Durk P Kingma and Yann Cun. Regularized estimation of image statistics by score matching. Advonces in neural information processing systems, 23, 2010.
Tomasz Korbak, Hady Elsahar, Mare Dymetman, and Germán Kruszewski. Energy-based models for code generation under compilability constraints. arXiv preprint S.S::2106.049560 2021.
Marc Lafon, Elias Ramzi, Clément Rambour, and Nicolas Thome. Hybrid energy based model the feature space for out-of-distributtin detection. arXie preprint arXiv:2305. 16966, 2023.
Yann LeCun. The mnist database of handwritten digits. hllp://yamn. lecun. com/endb/mmann 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujic Huang. A tutorial energy- based learning. Predicling struclured dala, 1(0), 2006.
Yukun Li and Liping Liu. Enhancing diffusion--ased point cloud generation with smoothness constraint. arXiv preprint arXiv:2404. 023966 2024.
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with energy-based models. Xir preprint arXiv:2102 00 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances neural information processing systems, 0-21464-211 2020.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 289 299-299,000 2023.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference on Compuler Vision and Pallern Recognation, pp. 2837 2845, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the TEEE/CVE conference on computer vision and pattern recognition, pp. 4460 4470, 2019.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceed- ings of the 28th international conference on machine learning (TCMI-11), pp. 1105-1112, 2011.
Anh-Tuan Nguyen and Sigrid Reiter. A performance comparison of sensitivity analysis methods for building energy models. In Building simulation, volume 8, pp. 651-664. Springer, 2015.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Frmon. Permutation invariant graph generation via score- based generative modeling. In International Conference on Artificial Intelligence and Statistics, pp. 4474-4484. PMLR, 2020.
Pingbo Pan, Ping Liu, Yan Yan, Tianbao Yang, and Yi Yang. Adversarial localized energy network for structured prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5347-5354 2020.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via finiteddifference score matching. Advances in Neural Informalion Processing Systems. 2020.
33:19175 19188, 1911917---988,,
Gabriel A Pinheiro, Johnatan Mucclini, Marinalva D Soares, Ronaldo C Prati, Juarez LF Da Silva, and Marcos G Quiles. Machine learning prediction of nine molecular properties based on the smiles represen- tation of the qm9 quantun-chemistry dataset. The Journal of Physical Chemistry A, 124(47):9854-9985) 2020.
Franco P Preparata and Michael I Shamos. Computational geometry: an introduction. Springer Science & Business Modia, 2012.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 352-660, 2017.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538 9551, 2022.
Jiaming Song, Arash Vahdat, Morteza, Mardani, and Jan Kautz. Pseudoinnerss--uuided diffussio models for inverse problems. In International Conference on Learning Representations, 2022.
Yang Song and Stelano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv:2101.0395 2021.
arXic preprint
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainly in Artificial Intelligence, pp. 574 584. PMLR, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprinl arXiv:2011.13456, 2020b.
Yang Song, Conor Durkan, Tain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415 1428, 2021.
Arash Vahdat and Jan Kautz. Nvac: A deep hicrarchical variational autoencoder. Advances neural information processing systems, 33:19667 19679,, 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-1674, 2011.
Martin J Wainwright, Michael I Jordan, al. Graphical models, exponential families, and variational inference. Foundations and Trends0 in Machine Learning, 1(1-2):1-30) 305,, 2008.
Li Wenliang, Danica J Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densitics. In International Conference on Machine Learning, pp. 6737-6746. PMLR, 2019.
Aoran Xiao, Jiaxing Huang, Dayan Guan, Xiaoqin Zhang, Shijian Lu, and Ling Shao. Unsupervised point cloud representation learning with deep neural networks: A survey. IEFF. Transactions on Pattern Analysis and Machine Intelligence, (5()):11321-1133) 2023. doi:
TTPAMM..023.3222772
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Generative voxelnet: learning energy-based models for 3d shape synthesis and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2468 2484, 2020.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Jianwen Xie, Yifei Xu, Zilong Zheng, Song- Zhu, and Ying Nian Wu. Generative pointnet: Deep energy- based learning on unordered point sets for 3d generation, reconstruction and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19776-1485,, 2021.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203. 02923, 2022.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training- energy- guided conditional diffusion model. arXiv preprint Stt.:23030098333 2023.
Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song- Chun Zhu, and Ying Nian Wu. Latent diffusion energy-based model for interpretable text modeling. aiv preprinl arXiv:2206.0558 2022.
Chengxi Zang and Fei Wang, Motlow: an invertible flow model for generating molecular graphs In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery 8 Data Mining, pp. 617 626, 2020.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang, Deep structured energy based models for anomaly detection. In International conference on machine learning, pp. 1100-1109. PMLR, 2016.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-inage translation via energy- guided stochastic differential equations. arXiv preprint orXin:2207.06635, 2022.
Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its application to texture modeling. Neural computation, 96)):1627-1660, 1997.
A Appendix
B Local minimum
Thcorem.
If n is at a local mazimum of the data log-likelihood, then Ep, [[()]] = 1 N EN T(x:).
Proof. Suppose we have N camples(#1,4d x2, ... NN and we want train our model via MLE.
First, the data likelihood can be written as
po. (x)
(17)
exp Σ Fo( + +n² EN T(x))
(m(0mm)
This leads to the log-likellnn ligeliiel as
= log
pe Pa. (x)
1=1
(18)
N
(x() + n
Σ
T(x;) n log Z(0, n)
i=1
13

---

Published in Transactions on Machine Learning Research (10/2024)
Then, we take the derivatives on both sides,
N T(x:) 2(0,n) m) On
0Z(0,n)
V ,,
(19)
2(0,7) .7) /
T(x;)
T(z)exp(Fo(()) +
(20)
N Σ
T(x:) - n po (T(x))
(21)
= 0, guaranteeing that the sample mean of the property function T(x) equals the expected value even we added one more constraint.
we can find the local minimum, then v,L
C Hyper-parameter sensitivity
We examined the impact various parameters on the model, with results shown Table and Table
We examined how the number of neural network layers affects the fitting point cloud data task. In Tab.
AT(a) denotes the difference between the distribution mean and the sample mean of the smoothness term. Our model is less sensitive to the hyper-parameleer and has smaller variance than the model without T(x) and has a smaller difference. Similar trends are observed in the task of fitting molecular data. In Tab. 6 Ar()) refers to the difference between the distribution mean and the sample mean of the valency term. The results show that as the number GNN layers increases, EDPGNN exhibits a higher variance in the validity ratio and Ar(x) whereas EDPGNN with ((x) shows minor changes.
Table 5: Effect of number of neural network layers
Latent diffusion model
Latent diffusion model (with T(x))
Number of neural network layers
Ar(x)
AT(x)
2
45
40.14
4
42
39
6
39.9
37.52
Table 6: Effect of the number of GNN layers
EDPGNN
EDPGNN (with T(x))
EDPGNN
EDPGNN (with T(x))
Number of GNN layers
Validity ratio (%)
Validity ratio (%)
At(x)
AT(x)
1
0.863
0.933
1.98
1.05
2
0.878
0.938
1.91
1.02
3
0.882
0.945
1.85
0.99
4
0.883
0.946
1.87
0.98
14


Reviews:
[{"review_id": "iAXM4RFXHi", "reviewer": "Reviewer_HrAQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although this paper is generally well-written and easy to follow, but I have several concerns on this paper and I cannot find enough strengths to accept this paper for this journal. I have described my concerns below.\n\n1. **Limited methodological contribution**\n    - This paper simply adds $\\eta^\\top\\mathbf{T}(\\mathbf{x})$ to the energy function, where $\\mathbf{T}(\\mathbf{x})$ is manually-designed statistic function based on inductive bias. In many cases, it is hard to design the function explicitly, which is why we use/learn a neural energy function. I think the proposed approach is not broadly applicable.\n    - As mentioned by the authors, there might be other technique to enforce a specific constraint (e.g., in molecule generation, atom-by-atom sequential generation can avoid cases where the valency constraint is violated). I'm not convinced why and when $\\mathbf{T}(\\mathbf{x})$ is superior to other techniques.\n    - Some functions cannot be (or are hard to be) applied to the intermediate sampling process: e.g., EBM sampling via Langevin dynamics and diffusion process. For example, the valency constraints in molecule generation cannot be utilized before atom's characteristic is determined (eq 14), and smoothness of the surface of $N$ points (eq 16) requires $O(N^2)$ operations to construct k-nn graph of the point cloud.\n    - In image generation, why is the statistic function (eq 15) required? I think the function limits the quality of generated samples. Can this function be generally applied in image generation?\n2. **Experimental results are not convincing**\n    - This paper conducts experiments with only a small set of benchmarks. In particular, MNIST benchmark is not enough to demonstrate the effectiveness of the proposed method in image generation tasks.\n    - The proposed method is not compared with a sufficient number of baselines. There are many generation frameworks for molecule generation, image generation, and point cloud generation. Therefore, I cannot be convinced that the proposed method improves the generation quality by a large margin.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper proposes a hybrid energy function that combines a neural energy function with a specially designed statistic function. This design choice allow us to utilize an inductive bias for each task.\n- This paper demonstrates the effectiveness of the proposed method in three applications: molecule generation, image generation, point cloud generation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Referring to Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gunDbb1gdg", "reviewer": "Reviewer_i2Kx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* Novel theoretical contribution on the hybrid model which combines energy based models with exponential family models.\n* Theoretically well grounded with past work clearly cited.\n\nWeakness:\n\n* Clarity of writing: Specifically some of the mathematical formulations in Section 4 which are hard to follow, especially for those not as well versed in this specific domain. Also in the subsections of 4.2 for each of the empirical statistic functions.\n\n* Ablation studies and scalability: While the early results are encouraging, it would be interesting to see the results on high dimensional widely adopted benchmarks to give s sense of scalability. Additionally, while this is partially addressed in  5.4 (Fig 1), it would be interesting to see the influence of various subcomponents of this hybrid model to the overall metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The given work proposes an energy based model which incorporates a neural energy function and novel parameter less statistic function.  The authors argue that this approach inherits property of exponential family models which allows them to express special statistics  in energy function as an inductive bias. The authors also present an empirical study on three tasks (molecule graph generation, image generation, point cloud generation)  to verify their claims of statistic matching .", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major points:\n* Clarity of writing around mathematical formulation and individual statistical functions for the empirical evaluation.\n* Hyper-parameter sensitivity and  empirical evaluation for higher dimensional datasets.\n\n\nMinor point:\n * Few grammatical corrections. For example \"Specifically, We augment\" -> \"Specifically, we augment\"  in line 5 of the abstract.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HGirSh2uYS", "reviewer": "Reviewer_s9Tb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The idea of this paper is simple yet effective: the proposed approach directly combines two existing methods, successfully incorporating known knowledge through the statistic function into the energy-based model.\n- The paper offers theoretical evidence for the proposed method, affirming that the hybrid approach maintains the advantages of utilizing the statistic function.\n- The paper includes some examples of statistic functions to illustrate the broad applicability of their approach.\n\n**Weakness**\n- I suggest that the explanation of theoretical results could be further improved by providing more specific details or citing related references to make it easier for a general audience to understand.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a hybrid approach that combines energy-based models with exponential family models to incorporate inductive bias in data modeling. By augmenting the energy function with a well-studied statistic function, the model captures relevant specified rules associated with the statistic function, providing inductive bias. Empirical studies confirm the effectiveness of the statistic-matching property when informative statistics are utilized.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Improvement suggestions for theoretical explanation:**\n- Enhance the explanation of the difference between the distribution mean and the data mean by utilizing Equation 9 and discussing its relation to $\\Delta_{T(x)}$ (where absolute values are taken).\n- Provide a clearer and more precise discussion on the convexity of the negative likelihood.\n\n**Typo corrections:**\n- *Abstract:* Specifically, \"we\" augment the energy term.\n- *Page 3:* A neural network with learnable parameters $\\theta$.\n- Change uppercase $L$ to lowercase$l$ in the statement of Theorem 1 as well as its proof.\n- Address the missing reference for \"[cite]\" where methods employing other techniques to enforce valency constraints are discussed.\n- The usage of the symbol $*$ in Equation 8 is not common, as it is typically associated with convolution.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a simple yet effective approach that incorporates inductive bias into energy-based models (EBMs) for data fitting. They demonstrate the effectiveness of the approach across three different domains: images, point clouds, and molecules. While one reviewer remains negative after the discussion, the majority of the reviewers, along with Action Editor (AE), agree that the claims made in the submission are well-supported by clear evidence. AE also believes that practitioners would find this approach valuable, as inductive bias is often known by domain experts in real-world applications, making the approach both applicable and capable of improving EBM performance.

Therefore, AE recommends acceptance but strongly encourages the authors to conduct additional experiments to further validate the effectiveness of the proposed approach.
Decision: Accept (poster)