Versatile Energy-Based Probabilistic Models for High Energy Physics
Authors: Taoli Cheng
Aaron Courville
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative modeling
Energy-based models
Out-of-distribution detection
Sciences
Application
Physics

Abstract: As a classical generative modeling approach, energy-based models have the natural advantage of flexibility in the form of the energy function. Recently, energy-based models have achieved great success in modeling high-dimensional data in computer vision and natural language processing. In line with these advancements, we build a multi-purpose energy-based probabilistic model for High Energy Physics events at the Large Hadron Collider.  This framework builds on a powerful generative model and describes higher-order inter-particle interactions. It suits different encoding architectures and builds on implicit generation. As for applicative aspects, it can serve as a powerful parameterized event generator for physics simulation, a generic anomalous signal detector free from spurious correlations, and an augmented event classifier for particle identification.

Versatile Energy-Based Probabilistic Models for High Energy Physics
Taoli Cheng Mila University of Montreal chengtaoli i. 19900gmail.com com
Aaron Courville Mila University Montreal aaron. wwtp::coooiiillleutteeeerrr
Abstract
As classical generative modeling approach, energy- models have the natural advantage of flexibility in the form the energy function. Recently, energy- energy-based bbssed models have achieved great success in modeling high-dimensional data in computer vision and natural language processing. In line with these advancements, we build a multi-purpose energy-based probabilistic model for High Energy Physies events the Large Hadron Collider. This framework builds on a powerful generative model and describes higher-order inter-particle interactions. It suits different encoding architectures and builds on implicit generation. As for applicative aspects, can serve powerful parameterized event generator for physics simulation, a generic anomalous signal detector free from spurious correlations, and an augmented event classifier for particle identification.
Introduction
The Large Hadron Collider (LHC) [1, the most energetic particle collider in human history, collides highly energetic protons to examine the underlying physics of subatomic particles and extend our current understanding the fundamental forces of nature, summarized by the Standard Model of particle physics. After the great success in observing the Higgs boson [8]] the most critical task of searching for new physies signals remains challenging. High Energy Physics (HEP) events produced at the LHC have the properties of high dimensionality, high complexity, and enormous data size. To detect rare signals from enormous background events, physics observables describing these patterns have been used to identify different types of radiation patterns. However, it's not possible to utilize all the information recorded by the detectors with a few expert-designed features. Thus deep neural classifiers and generative models, which can easily process high-dimensional data meet the needs for more precise data analysis and signal detection.
Energy-based Models (EBMs) [38 3 46). as a classical generative framework, leverage the energy function for learning dependencies between input variables. With an energy funccion E(x) and constructing the un-normalized probabilities through the exponential p(x) exp(-E the energy model naturally yields probability distribution. Despite the flexibility in the modeling, the training of EBMs has been cumbersome and unstable due to the intractable partition function and the corresponding Monte Carlo sampling involved. More recently, EBMs have been succeeding in high-dimensional modeling [55] [6 25
51 for computer vision and natural language processing, the same time, it has been revealed that neural classifiers are naturally connected with FBMs 67 33 32), combining the discriminative and generative learning processes in common learning regime. More interestingly, compositionality (30) can be easily incorporated within the framework of EBMs by simply summing up the energy functions [2]] 22). On the other hand, statistical physics originally inspired the invention of EBMs. This natural connection in formalism makes EBMs appealing modeling physical systems. In physical sciences, EBMs have been used to
23
62
19
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Topic
Practice
Generative modeling
Parameterized event generation Model-indeeendenn new physics search
Out-of-distribution detection
Hybrid modeling
Classifier combined with EBMs
Table I: Application aspects of Energy-based Models for High Energy Physics.
simulate condensed-matee systems and protein molecules 57 They have also been shown great potential in structure biology [24 in use case of protein conformation.
Motivated by the flexibility in the architecture and the compatibility with different tasks, we explore the potential of EBMs in modeling radiation patterns of elementary particles at high energy. The energy function is flexible enough to incorporate sophisticated architectures. Thus EBMs provide a convenient mechanism to simulate complex correlations in input features or high-order interactions between particles. Aside from image generation, applications for point cloud data [68]. graph neural networks [48] for molecule generation are also explored. In particle physics, we leverage the self-attention mechanism 6 65), to mimic the complex interactions between elementary particles.
As one important practical application, neural net-based unsupervised learning of physics events 42 g 64 have been explored in the usual generative modeling methods including Generative Ad- versarial Networks (GANs) [1] and Variational Autoencoders (VAEs) [45). However, GANs employ separate networks, which need to be carefully tuned, for the generation process. They usually suffer from unstable training, high computation demands, and mode collapse. In comparison, VAEs need well-designed reconstruction loss, which could be difficult for sophisticated network architectures and complex input features. EBMs thus provide strong alternative generative modeling framework of LHC events, by easily incorporating sophisticated physics -issii-d neural net architectures.
[8
At the same time, EBMs can serve as generic signal detectors, since out-of-distributioo (OOD) detection comes naturally in the form of energy comparison. More importantly, EBMs incur fewer spurious correlations in OOD detection. This plays slightly different role in the context of signal searches at the LHC. There are correlations that are real and useful but at the same time hinder effective signal detection. As we will see in Section 4.2 the EBMs are free from the notorious correlation observed in many anomaly detection methods in HEP, in both the generative and the discriminative approaches.
As summarized in Table we build a multi-tasking framework for High Energy Physics. To that end, we construct energy-based model of the fundamental interactions of elementary particles to simulate the resulting radiation patterns. We especially employ the short-run Markov Chain Monte Carlo for the EBM training. We show that EBMs are able to generate realistic event patterns and can be used as generic anomaly detectors free from spurious correlations. We also explore EBM- based hybrid modeling combining generative and discriminative models for HEP events. This unified learning framework paves for future- future-genert generation event simulators and automated new physics search strategies. It opens door for combining different methods and components towards a powerful multi-tasking engine for scientific discovery at the Large Hadron Collider.
2 Problem Statement
Describing HEP Events Most particle interactions happening at the LHC are governed by Quantum Chromodynamics (QCD), due to the hadronic nature of the proton-proton collision. Thus jets are enormously produced by these interactions. A jet is formed by collimated radiations originating from highly energetic elementary particles (c.g., quarks, gluons, and sometimes highly-boosted electro-weak bosons). The tracks and energy deposits of the jets left in the particle detectors reveal the underlying physics in complex patterns. A jet may have hundreds of jet constituents, resulting in nigh-dimensional and complex radiation patterns and bringing difficulties in encoding all the information with expert features. Reconstructing, identifying, and classifying these elementary particles manifest in raw data is thus critical for ongoing physics analysis at the LHC.
Specifically, each constituent particle within a jet has (logPr, 7, Φ) as the descriptive coordinates in the detector's 's reference frame. Here, Pr represents the transverse momentum perpendicular to the

---

beam axis, and (1), d) denotes the spatial coordinates within the cylindrical detector. (More details about the datasets can be found in Appendix(AA] Thus jet can be described by the high-dimensional vector x [(log pr, 71 e); J." supposing the jet has N consttttunts. Our goal is thus to model the data distribution p(x) precisely, and p(y) x) in the case of supervised classification with y denoting the corresponding jet type.
Parameterized Generative Modeling At the LHC, event simulation serves as an important handle for background estimation and data analysis. For many years, physics event simulators [10) have been built on Monte Carlo methods based on physics rules. These generators are slow and need to be tuned to the data frequently. Deep neural networks provide with an efficient parameterized generative approach to event simulation for the coming decades. Generative modeling in LHC physics has been experimented with GANs immage-based [58] (7] and point-cloud- based (43)) and VAEs [64]. There are also GANs [ working on high-level features for event selection. However, as mentioned in the introduction, these models all require explieit generators, which brings obstacles to situations with complex data formats and sophisticated neural architectures.
OOD Detection for New Physics Searches Despite the great efforts in searching for new physics signals at the LHC, there is no hint of beyood-Standard Model physics. Given the large amount of data produced at the LHC, it has been increasingly challenging to cover all the possible search channels. Experiments at the LHC over the past decades have been focused on model-oriented searches, as in searching for the Higgs boson (26] 36 The null results up to now from model-driven searches call for novel solutions. We thus shift to model-independent and data-driven anomaly detection strategies, which are data-oriented rather than theory- for new physics searches.
Neural bass- anomaly detection in HEP takes different formats. Especially, unsupervised generative models trained background events be used to detect potential unseen signals. More specifically. Variational Autoencoders 15
41 5 have been employed to detect novel signals, directly
based on detector- features. High-level observables-basee VAEs 12 39 have also been explored. However, naively trained VAEs (and Autoencoders) are usually subject to spurious correlations which result in failure modes in anomaly detection, partially due to the format of the reconstruction loss involved. Usually, one needs to explicitly mitigate these spurious correlations with auxiliary tasks such as outlier exposure 35 [5]. or modified model components such as autoencoders with smeared pixel intensities [29] and autoencoders augmented with energy-based density estimation [69 20)..
At the same time, EBM naturally has the handle for discriminating between in an-distribution and out- of-distribution examples and accommodates tolerance for spurious correlations. While in-distribution data points are trained have lower energy, energies of OOD examples are pushed up in the learning process. This property has been used for OOD detection in computer vision [25] [3 23 71. This thus indicates great potential for EBM-ba new physics detection at the LHC.
3 Methods
3.1 Energy Based Models
Energy-based models are constructed model the unnormalized data probabilities. They leverage the property that any exponential term exp(-E(xx) is non-negative and thus can represent unnormalized probabilities naturally. The data distribution is modeled through the Boltzmann distribution: po(x) = xxp(=F((x) (0) with the energy model (x X + R mapping x E X to a scalar. And the partition function& Z- [( = exp(p Fxx(x) (x]]dx ddx integrates over all the possible states.
EBMs can be learned through maximum likelihood Ep (log pe (x)). However, the training of EBMs can pose challenges due to the intractable partition function log pe()) Though the partition function is intractable, the gradients of the log-likelihood do not involve the partition function directly. Thus when taking gradients w.r.t. the model parameters 8, the partition
Eig (x) log (0).
We note that this energy function not to be confused with the physical energy of the HEP events/objects. Here the energy is essentially an abstract quantity used for probabilistic modelling of the radiation patterns and substructures. It can approximately serve as measure of radiation substructure or complexity when describing HEP events.
3

---

MCMC Samples
Random Noise
Xg ~ (x(x)
* ( Ø)
MLP
Gradient-based Markov Chain
Monte Carlo
Pooling
d. (o t
Ador
p(s,a)
Food Forward
Contrastive Divergence
20(x) [x(xx) Bps(x) Bo(x
Addi/SMomm))
EB Ex(x,8) 3)
Mutt-Heed Altersioon
Embeddingg
Data
(batch: sizc, Ion 500,
{(lggrr n d).)
Figure 1: Schematic of the EBM model. The energy function E(x, y) is estimated with a transformer. The training procedure is governed by Contrastive Divergence (the vertical dimension), for which the model distribution estimation go(x) (x) is obtained with Langevin Dynamics (the horizontal dimension), evolving samples from random noises X
function is canceled out. The gradient of the maximum likelihood loss function be written as:
VoC(O) Ep (x) Vo (x
Ex (x) VoLis Exx <xp((x) [o
(2)
where PD (x) is the data distribution and (x) is the model distribution. The training objective is thus composed of two terms, corresponding to two different learning phases (i.c., the positive phase fit the data x+, and the negative phase to fit the model distribution x-). When parameterizing the energy function with feed-forward neural networks [54], the positive phase is straightforward. However, the negative phase requires sampling
the model distribution. This leads various Monte Carlo sampling strategies for estimating the maximum likelihood.
Contrasting the energies of the data and the model samples as proposed Contrastive Divergence (CD) 37
leads to an effective strategy to train EBMs with following CD objective:
DKL. (x)((p)))loo
DRL. (TI((PpD) (Tpn (oo))loox)))) po()),
where 7 denotes the one-step Monte Carlo Markov Chain (MCMC) kernel imposed on the data distribution. In more recent approaches for high-dimensional modeling [56] 25], we can directly initialize from random noises to generate MCMC samples instead of initializing from the data distribution as the original approach. This strategy also helps with exploring and mixing between modes.
Negative Sampling The most critical component in the negative phase the sampling to estimate the model distribution. We employ gradient- based MCMC generation for the negative sampling, which handled by Langevin Dynamics [52] (66) As written Eq, 4 Langevin dynamics uses gradients w.r.t. the input dimensions, associated with diffusion term to inject stochasticity, to generate a sequence negative samples (xx K
1k=1"
22 Ex A A-e, with ((0
X X
(4)
2
MC Convergence convergent MCMC shows that short-run (~ 5- 5-000 steps) MCMC initialized from random distributions
The training anatomy [55
56 for short-run non-convergent MCMC and long-run

---

is able to generate realistic samples, while long-run run MCMC might be oversaturated with lower-energy states.
To improve mode coverage, we use random noise to initialize MCMC chains. To accelerate training, employ a relatively small number of MCMC steps. In practice, we reuse the generated samples as initial samples of the following MCMC chains to accelerate mixing, similar to Persistent Contrastive Divergence [63]. Following the procedure in [25), we use a randomly initialized buffer that is consistently updated from previous MCMC runs as the initial samples. (As empirically shown 13 66]. Metropolis-Hastingg step is not necessary. So we ignore this rejection update in our experiments.)
Energy Function Since there is no explicit generator in EBMs, we have much freedom in designing the architectures of the energy function. This also connects with the fast-paced development of supervised neural classifier architectures for particle physics. We can directly reuse the architectures from supervised classifiers in the generative modeling of EBMs. We use self-attention-base transformer to parameterize the energy function E.).. We defer the detailed description to Sec. 3.3
The full algorithm for training EBMs is described in Algorithm
Algorithm EBM training with MCMC by Langevin Dynamics
Input: training samples {x,- J from pD(x), (x), parameterized energy function Ea(-), initial buffer B + 23, Langevin dynamics step size Ax. number of MCMC steps K, model parameter learning
rate λg. regularization strength a for Gradient descent step = O...L-I do
x
(x)
x 0.95 B + 0.05
Reinitialize the samples in the buffer with random noise in the
probability of 0.05
for Langevin dynamics step k= 0-1 do
x,,+1 = i.A Xx V F.o 0.005 6k, ~ N/0, 1) Langevin Dynamics taking
gradients w.r.t. input dimensions
end for
X <- x,KK
CCD 1 E,
(x;²) ))
Lreg N E (Eg(x)) Eo xx ))²
L2 Regularization
I + 0 Vo &Lrreg)
Update model parameters with gradient descent
B t X,KK UB
Update the buffer with generated samples
end for
3.2 Hybrid Modeling
Neural Classifier as an EBM
A classical classifier can be re-interpreted in the framework of 40). with the logits g(x) corresponding to negative energies of the joint
EBMs
46
33
67
17
distribution p(x, 3)
(xp(g(x),)
Z where g(x), denotes the logit corresponding to the label y. Thus
the probability marginalized over y can be written as p(x) Σ, exp(R(x)x) with the energy of x as
exp(g(x),) llog E, exp(g(x),). We are then brought back to the classical softmax probability Σ, (x)) when calculating p(y|x).
This viewpoint provides a novel method for jointly training supervised classifier and an unsupervised generative model. Specifically, (33) successfully incorporated EBM--ased generative modeling into a classifier in image generation and classification. We follow their proposal to train the hybrid model follows to ensure the classifier p(y]x) is unbiased. The joint likelihohoood is decomposed into two terms:
logp(x, x, v) = logp(x) + log]())x)
(5)
Thus one can maximize log p(x) with the contrastive divergence of the EBM with the energy function E(x) log Σ, exp(g(x),), and maximize log p(y(x) with the usual cross-entropy of the classification.
5

---

3.3 Energy- Models for Elementary Particles
We would like to construct an energy- basey model for describing jets and their inner structures. In conceiving the energy function for these elementary particles, we consider the following constraints and characteristics:
permutation invariance the energy function should be invariant jet constituent permutations, and 2) higher-order interactions we would like the energy function to be powerful enough to simulate the complex inter-particle interactions.
Thus, we leverage the self-attention-hase transformer [65] to approximate the energy function, which takes into account the higher-order interactions between the component particles. As indicated in Eq. [6b] the encoding vector of each constituent W is connected with all other constituents through the sell-attention weights A in Eq. 6al which are already products of particle representations Q and K.
A softmax(Q KJ //v Vdmodel)
(6a)
W = A
(6b)
Moreover, we easily incorporate particle permutation invariance 70 in the transformer, by summing up the encodings of each jet constituent. The transformer architecture is shown in Fig. The coordinates (log PT, 17, o) of each jet constituent are first embedded into dmmota-dimeensonal space through a linear layer, then fed into Nr, self-attention blocks sequentially. After that, a sum- pooling layer is used to sum up the features of the jet constituents. Finally, a muult-layer--pereepe projector maps the features into the energy score. Model parameters are recorded in Table of Appendix[A.
4 Experiments
Training Details
The training set consists of 300,000 QCD jets. We have 10,000 samples in the buffer and reinitialize the random samples with a probability of 0.05 in each iteration. We use relatively small number of steps (e.g.. 24) for the MCMC chains. The step size Xx is set to 0.1 according to standard deviations of the input features. The diffusion magnitude within the Langevin dynamics is set to 0.005. The number of steps used in validation steps is set to 128 for better mixing.
We use Adam 44 for optimization, with the momenta 31 0.0 and B2 = 0.999. The initial learning rate is set to le-4, with a decay rate of 0.98 for each epoch. We use batch size of 128, and train the model for 50 epochs. More details can be found in Appendix
Model Validation In monitoring the likelihood, the partition function can be estimated with An- nealed Importance Sampling (AIS) [53). However, these estimates be erroneous and consume lot of computing resources. Fortunately for physics events, we have well-designed high-level features as a handle for monitoring the generation quality. Especially, we employ the boost-invariant jet transverse
momentum pr EN PT; and the Lorentz-invariant jet mass M V N E, (E.") p.)²
as the validation observables. And we calculate the Jensen-Shannon Divergence (ISD), between these high-level observable distributions of the data and the model generation, the metric. In contrast to the short-run MCMC in the training steps, we instead use longer MCMC chains for generating the validation samples.
When we focus on downstream tasks such as OOD detection, it's reasonable to employ the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) as the validation metric, with Standard Model top jets serving as the benchmark signal.
Generation Test-time generation is achieved in MCMC transition steps from the proposal random (Gaussian) distribution. We use a colder model and a smaller step size at test time which is annealed from 0.1 and decreases by factor of 0.8 in every 40 steps to encourage stable generation. The diffusion noise magnitude is set to 0.001. Longer MCMC chains with more steps (e.g.. 200) are taken to achieve realistic generation.
consistent observation across various considered methods is that the step size stands out as the crucial parameter predominantly influencing the quality of generation.
IIn practice, MCMC-based generation could be slow, especially in cases with very long MCMC chains. Though in our case, 200 steps are enough to produce very realistic samples, it is worth exploring methods to accelerate the generation process.

---

Anomaly Detection In contrast to prevalent practices in computer vision, EBM- OOD detection in LHC physics exhibits specificity. The straightforward approach of comparing two distinct datasets, such as CIFAR I and SVHN, overlooks the intricate real-world application environments. Adapting OOD detection to scientific discovery at the L.H,, we reformulate and tailor the decision process as follows: if we train on multiple known Standard Model jet classes, we focus on class- conditional model evaluation for discriminating between the unseen test signals and the most copious background events (i.e., QCD jets rather than all the Standard Model jets),
4.1 Generative Modeling - E-neryy-based Event Generator
We present the generated jets transformed from initial random noises with the Langevin dynamics MCMC. Due to the www--hmma--reddabe nature of physics events (e.g., low-level raw records at the particle detectors), we are not able to examine the generation quality through formats such as images directly. However, it has a long history that expert-desinned high-level observables can serve as strong discriminating features. In the first row of Fig. 2 we first show the distributions of input features for the data and the model generation. Meanwhile, in the second row, we plot the distributions of high-level expert observables including the jet transverse momentum pr and the jet mass M. Through modeling low-level features in the detector space, achieve precise recovery of the high-level physics observables in the theoretical framework. For better visualization, we can also map the jets onto the (1, Φ) plane, with pixel intensities associated with the corresponding energy deposits. We show the average generated jet images in Fig. of Appendix B comparing to the real jet images (right-most) in the (n, Φ) plane. In Table 2 we present the Jensen-Shannon Divergence of the high-level observables pr and M distributions between real data and model generation, as the quantitative measure of the generation performance.
0.35
Medei Geevrattion
Bath
5-
Bata
Model Generation
4.0
Dath
Midsi Glevration
0.3
825
2.5-
0.20
#3
2.0-
0.15
0.10
0.05
0.00
00.5
3.0
0.5
-0.5
0.0
0.5
1.0
DnV
1.077
1.04
040
0.08
0.5%
0.00
p. 2.1661
Figure 2: Top: Input feature distributions of jet constituents for the data and the model generation. Bottom: High-level feature distributions for the data and the model generation.
Model
JSD (PT) 10-1
JSD 10-/10-1
JSD (M/pr) 10
8-VAE 15 EBM
3.7
11.0
2.4 + 1.9
2.0
6.3 + 4.6
Table Model comparison in terms generation quality measured in of high-level observables pr and M. For EBMs, we present the means and standard deviations for the JSDs obtained from independent runs. We also present a 3-VAE [15) model baseline.
Divergence
7

---

4.2 Anomaly Detection Anomalous Jet Tagging
Since EBMs naturally provide an energy score for each jet, for which the in-distribution samples should have lower scores while OOD samples are expected to incur higher energies. Furthermore, classifier, when interpreted as an energy-based model, the transformed energy score can also serve as an OOD identifier R3 49
EBM In HEP, the in-situ energy score can used to identify potential new physics signals. With an energy-based model, which is trained the QCD background events or directly the slightly signal-contaminated data, we expect unseen signals (i.e., non-QCD jets) to have higher energies and correspondingly lower likelihoods.
In Fig. 3] we compare the energy distributions of in-distribution QCD samples, -ittr-ddiittibttionn signal examples (hypothesized heavy Higgs boson 7 which decays into four QCD sub-jets), and random samples drawn from the proposal Gaussian distribution. We observe that random samples unusually have the highest energies. Signal jets have relatively higher energies compared with the QCD background jets, making model-independent new physics searches possible.
Spurious Correlation A more intriguing property of EBMs is that spurious correlations can be better handled, Spurious correlations in jet tagging might result in distorted background distributions and obscure effective signal detection. For instance, VAEs in OOD detection can be highly correlated with the jet masses [5], similar to the spurious correlation with image pixel numbers in computer vision 150 60]. In the right panel of Fig. B plot the correlation between the energy score and the jet mass. Unlike other generative strategies for model- indeleededded anomaly detection, EBMs largely free from the spurious correlation between the energy E(x) and the jet mass M. The underlying reason for EBMs not presenting mass correlation could be the negative sampling involved in the training process. Larger mass modes are already covered during the negative sampling process. This makes EBMs a promising candidate for model-ine mineel-independent new physics searches.
DCD
Signal
10.
Rindom
Randon
-1-
-2-
-2
400
MyGeV
Figure 3: Left: Energy distributions for random samples, background QCD jets, and novel signals. Right: Correlation between the jet mass M, and the energy F.
EBM-CLF The task of classifyin/fideniffiing different Standard Model jet types and the task of searching for beyond the Standard Model signals actually can be unified in single approach with neural classifiers distinguishing different Standard Model particle types [66. Compared with naive generative model based OOD, training neural classifiers on different known Standard Model jets helps the model with learning more meaningful and robust representations for effectively detecting new signals. Further combining generative models and discriminative classifiers can be achieved in the EBM framework elegantly. We employ the hybrid learning scheme (54 3 combining the discriminate and the generative approaches. It links with many OOD detection techniques and
observations. For instance, the maximum logit for OOD detection lowest energy miny Exx, y) in this framework.
34 comes in the format of the
8

---

We train an EBM-based multi-class classifier (EBM-CLF) according to Eq. 5 for both discriminating
different Standard jetsl (QCD jets, boosted W jets, and boosted top jets) and generating real-like like
jet samples. The results of generative sampling are shown in Appendix B The jointly trained EBM and jet classifier maintain the classification accuracy (see Appendix B). The associated EBM is
augmented by the discriminative task, and thus assisted with better inductive biases and domain knowledge contained in the in-distribution classes. EBM-CLF is an example of how we unify different physics tasks (jet classification, anomaly detection, and generative modeling) in unified framework.
We measure the OOD detection performance in the ROC curve and the AUC of the binary clas- sification between the background QCD samples and the signal jets. Table records the AUCs of different models (and different anomaly scoring functions) in tagging Standard Model Top jets and hypothesized Higgs bosons (OOD H). The jointly trained model generally has better anomaly tagging performance compared with the naive EBM. We also explore the norm of the score function of (x):: VV log (o || = Fxx]] serving as the anomaly score (similar to the approximate Constructed from the energy function, the score function approximately inherits the nice property of mass decorrelation. However, they have slightly worse performance compared with Exx) The corresponding ROC curves are shown in the left panel of Fig. 4 in terms of the signal efficiency es (i.e., true positive rate) and the background rejection rate I/EB (i.e., 1/false positive rate). In the right panel, we plot the background mass distributions under different cuts on the energy scores. We observe excellent jet mass decorrelatiioninvaiiin anomalous jet tagging. Additionally for EBM-CLF, we also record the AUCs for anomaly scores of the condiionditi softmax probability p(y(x) and the logit g(x), y E(x, y) corresponding to the background QCD class, in Table However, without further decorrelation strategies (auxiliary tasks or assistant training strategies), these two anomaly scores usually strongly correlated with the masses of the in-distribution classes and distort the background distributions. Thus we list the
mass in 33
for energy score- based
results here only for reference.
Model
AUC (Top)
AUC (OOD H)
DisCo- VAE (R- = 1000) 15]
0.593
0.481
KL-OE-VAB 15
0.744
0.625
dass-decoreelated
EBM (E(x))
0.679 = 0.009
0.794 = 0.032
EBM V log (x))l)
0.628 = 0.017
0.738 + 0.033
EBM-CLF (E))))
0.703 + 0.020
0.815 + 0.018
EBM-CLF ( V log po po (I)
0.679 + 0.040
0.722 + 0.028
Mass- cars-coreelated
EBM-CLF (g(),,) EBM-CLF (p(y)x)
0.920 + 0.002
0.877 + 0.008
0.940 + 0.002
0.865 + 0.012
Table 3: Anomaly detection performance for different models (anomaly scores in parentheses) measured AUCs. We present the averaged AUCs over 5 independent models with different random seeds and the associated standard deviations. We list a few baseline models with mass decorrelation achieved either by regularization term IDisCo-VAE (x- 1000)) or an auxiliary classification task contrasting in-distribution data and outlier data VEL-OE-VAE)
5 Conclusion
We present a novel energy-based generative framework for modeling the behavior of elementary particles. By mimicking the inter-particle interactions with sil-attenniion-asee transformer, we map the correlations in the detector space to probabilistic space with an energy function. The energy model is used for the implicit generation of physics events. Despite the difficulty in training EBMs, we adapted the training strategies to balance learning efficiency and training stability. We adopt short-rum MCMC sampling at training, while at test time we instead use dynamic generation to obtain prominent generative quality. Additionally, the framework supports the construction of an augmented classifier with integrated generative modeling. This unified approach provides us with flexible tools for high-performance parameterized physics event simulation and muurious-correlation free modee--indeenndent signal detection. It marks a significant stride in nene-genno multitasking machine learning models for high-energy physics.
9

---

BEMCEDOL. AUCITap>~0.05
OCD
BEMIEIOI. AUCIR)-0,724
10"2
QC3
EBM CLPIEOI), (-0,70
0C0 0.2]
10
REMGCLFINNS) AUC01)-6.815
OCD [x- 0.11
10
1/En
ratalom
10-4
10a
10
100
Es
MyGGVV
Figure 4: Left: ROC Curves for EBM and EBM-CLF with the energy E(x) as the anomaly score. The grey line denotes the case of random guessing. Right: Background mass distributions under different acceptance rates after cutting on the energy score from the EBM-CLF.
Acknowledgments and Disclosure of F Funding
This work is partially supported by the IVADO Postdoctoral Research Funding. We acknowledge the computing resources provided by Mila (mila.quebec).
References
[1] L.H. Machine. JINST, 3:S08001, 2008.
[2] Georges Aad et al. Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B, 716:1-29, 2012.
[3] David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, (1):147-166, 1985.
[4] Johan Alwall, Michel Herquet, Fabio Maltoni. Olivier Mattelaer, and Tim Stelzer. Madgraph going beyond. Journal of High Energy Physics, 2011 Jun 2011.
[5] Oliver Atkinson, Akanksha Bhardwaj, Christoph Englert, Partha Konar, Vishal S. Ngairangbam, and Michael Spannowsky. IRC-Safe Graph Autoeneoder for Unsupervised Anomaly Detection. Front, Artiff Intell., 5:943135, 2022.
[6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, 0D/1140900433 2015.
[7] G.C. Branco, P.M. Ferrcira, L. Lavoura, M.N. Rebelo, Mare Sher, and Joao P. Silva. Theory and phe- nomenology of wwo-Higgy-ooublet models. Phys. Rept., 516:1-102, 2012.
Anja Butter, Tilman Plehn, and Ramon Winterhalder. How GAN LHC Events. SciPost Phys., 7(6):075. 2019.
[9] Matteo Cacciari, Gavin P Salam, and Gregory Soyez. The anti-ktjj jet clustering algorithm. Journal of High Energy Physics, 2008(04):063 063, Apr 2008.
[10] J. M. Campbell et al. Event Generators for High-Energy Physies Experiments. In 2022 Snowmass Summer Study, 2022.
[11] Miguel Á. Carreira- Pereira-errpifán and Geoffrey Hinton. contrastive divergence learning, In Robert Cowell and Zoubin Ghahramani, editors, Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, volume R5 of Proceedings of Machine Learning Research, pages 33-40. PMLR. 06 06-08 08 Jan 2005, Reissued PMLR on 30 March 2021.
10

---

[12] Olmo Cerri, Thong Q. Nguyen, Maurizio Pierini, Maria Spiropulu, and Yean-Roch Vlimant. Variational Autoencoders for New Physics Mining the Large Hadron Collider. JHEP, 05:036, 2019.
[13] Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference on Machine Learning, 2014.
[14] Taoli Cheng. Test sets for jet anomaly detection at the lhe, March 2021.
[15] Taoli Cheng, bean-François Arguin, Julien Leissner-Martin, Jacinthe Pilette, and Tobias Golling. Variational autoencoders for anomalous jet tagging. Phys. Rev. D. 107(1):01002, 2023.
[16] Taoli Cheng and Aaron Courville. Invariant representation driven neural classifier for anti Aadi-QCDup jet tagging. JIEEP, 10: 152, 2022.
[17] Jifeng Dai and Ying Nian Wu. Generative modeling of convolutional neural networks. CoRR. abs/1412.6296,, 2014.
[18] J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaitre, Mertens, and M. Selvaggi. Delphes 3: a modular framework for fast simulation of generic collider experiment. Journal of High Energy Physics, 2014(2), Feb 2014.
[19] Yuntian Deng. Anton Bakhtin, Myle Ou, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. In International Conference on Learning Representations, 2020.
[20] Barry M. Dillon, Luigi Favaro, Tilman Plehn, Peter Sorrenson, and Michael Krämer. A normalized autoencoder for lhe triggers, 2022.
[21] Yilun Du Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models.
[22] Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In NeurIPS, 2021.
[23] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models.
[24] Yilun Du, Joshua Meier, Jerry Ma. Rob Fergus, and Alexander Rives. Energy-based models for atomic- resolution protein conformations. In International Conference on Learning Representations, 2020.
[25] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models.
[26] F. Englert and R. Brout. Broken symmetry and the mass of gauge vector mesons. Phys. Rev. Leut 13:321-323, Aug 1964.
[27] Martin Erdmann, Jonas Glombitza, and Thorben Quast. Precise simulation of electromagnetic calorimeter showers using Wasserstein Generative Adversarial Network. Comput. Softw: Big Sci., 3(1):4, 2019.
[28] S. Chatrchyan et al. Observation of new boson at mass of 125 gev with the ems experiment at the lhe. Physics Letters B, 716(():30-61, 2012.
[29] Thorben Finke, Michael Krämer, Alessandro Morandini, Alexander Mück, and Ivan Oleksiyuk. Autoen- coders for unsupervised anomaly detection in high energy physies. JHEP, 06:161, 2021.
[30] J.A. Fodor and E. LePore. The Compositionality Papers. Clarendon Press, 2002.
[31] Tan Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications the ACM, 63(11):139-
144, 2020.
[32] Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Kristjanson Duvenaud. No meme for me: Amortized sampling for fast and stable training of energy-based models. ArXiv, abs/2010.04230, 2021.
[33] Will Grathwohl, Kuan-Chich Wang, Jörn- Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one.
[34] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Xiaodong Song. Scaling out-of -fi-t-i-istiittii detection for real-world settings. In International Conference Machine Learning, 2022.

---

[35] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2019.
[36] Peter W. Higgs. Broken symmetries, massless particles and gauge fields. Phys. Lett., 12:132 133, 1964.
[37] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8):1771-1800, aug 2002.
[38] Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):255--255 1982.
[39] Pratik Jawahar, Thea Aarrestad, Nadezda Cheryyaskkaya, Maurizio Pierini, Kinga A. Wozniak, Jennifer Ngadiuba, Javier Duarte, and Steven Tsan. Improving Variational Autoencoders for New Physics Detection at the LHC With Normalizing Flows. Front. Big Data, 5:803685, 2022.
[40] Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In
I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[41] Alan Kahn, Julia Gonski, Inès Ochoa, Daniel Williams, and Gustaaf Brooijmans. Anomalous jet identifica- tion via sequence modeling. JINST, 16(08):P08012, 2021.
[42] Raghav Kansal, Javier Duarte, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Roch Vlimant, and Dimitrios Gunopulos. Graph Generative Adversarial Networks for Sparse Data Generation in High Energy Physics. In 34th Conference Neural Information Processing Systems, 11 2020.
[43] Raghav Kansal, Javier Duarte, Hao Su, Breno Orzari, Thiago Tomei, Maurizio Pierini. Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopulos. Particle Cloud Generation with Message Passing Generative Adversarial Networks. 6 2021.
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[45] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.611 11,,1111422 2014.
[46] Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, and Jie Huang. A tutorial on energy-based learning. 2006.
[47] Julien Leissner-Martin Taoli Cheng, and Jean-Francois Arguin. Qcd jet samples with particle flow constituents, July 2020.
[48] M. Liu, Kegiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with energy-based models. ArXiv, abs/2102.00546, 2021.
[49] Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Proceedings of the 34th International Conference on Neural Information Processing Syxtems, NIPS'20, Red Hook, NY, USA, 2020, Curran Associates Inc.
[50] Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayaman. Do deep generative models know what they don't 't know? ArXiv, abs/1810000 810.0911,, 2019.
[51] Subhajit Naskar, Pedram Rooshenas, Simeng Sun, Mohit Tyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. ACI, 2021.
[52] Radford Neal. MCMC Using Hamiltonian Dynamics. In Handbook of Markov Chain Monte Carlo, pages 113-162. 2011.
[53] Radford M. Neal. Annealed importance sampling. Statistics and Computing, 11:125-139, 2001.
[54] Jiquan Ngiam, Zhenghao Chen, Pang Wei Koh, and Andrew Y. Ng. Learning deep energy models. In Proceedings of the 28th International Conference International Conference Machine Learning. ICMI:11, page 1105-1112, Madison, WT, USA, 2011. Omnipress.
[55] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In AAAI, 2020.
[56] Erik Nijkamp, Mitch Hill, Gong-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run meme toward based model. In NeurIPS, 2019.
[57] Frank Noé, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365, 2019.
12

---

[58] Michela Paganini, Luke de Oliveira, and Benjamin Nachman. Accelerating Science with Generative Adversarial Networks: An Application 3D Particle Showers in Multilayer Calorimeters, Phys. Rev. Lett., 120(4):44203,, 2018.
[59] Huilin Qu and Loukas Gouskos. ParticleNet: Jet Tagging via Particle Clouds. Phys. Rev. D, 101(5)056019, 2020.
[60] J. Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan, Likelihood ratios out-of-distribution detection. ArXiv, abs/1900. 02845, 2019.
[61] Torbjöm Sjöstrand, Stephen Mrenna, and Peter Skands. brief introduction pythia 8.1. Computer Physics Communications, 178(11):855) (55(111)88-887 Jun 2008.
[62] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information P Processing Systems, volume 33, pages 12438 12438-12448. Curran Associates, Inc., 2020.
[63] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, page 1064-1077 New York, NY, USA, 2008. Association for Computing Machinery.
[64] Mary Touranakou, Nadezda Chernyavskaya, Javier Duarte, Dimitrios Gunopulos, Raghav Kansal, Breno Orzari, Maurizio Pierini, Thiago Tomei, and Jean-Rood Vlimant. Particle-based fast jet simulation at the lhe with variational autoeneoders, 2022.
[65] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. ArXiv, abs/1706.03362,, 2017.
[66] Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML'11, 11, page 681-688 168,, Madison, WI, USA, 2011. Omnipress.
[67] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. International Conference on Machine Learning, pages 6635-2644 PMLR, 2016.
[68] Jianwen Xie, Yifei Xu, Zilong Zheng, Song Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14971-14980, 2021.
[69] Sangwoong Yoon, Kung-Kyun Noh, and Frank Chongwoo Park. Autoencoding under normalization constraints. In International Conference on Machine Learning, 2021.
[70] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alex Smola. Deep sets. In NIPS, 2017.
[71] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection, In International Conference on Machine Learning, 2016.
13

---

A Experimental Details
Datasets For the standard EBM, we train on 300,000 simulated QCD jets. For the hybrid model EBM-CLE. we train on 300,000 simulated Standard Model jets (100,000 QCD jets, 100,000 boosted jets originating from the W boson, and 100,000 boosted jets originating from the top quark). For OOD detection the hypothetical Higgs boson (in the decay mode of + hh + (bb) with mass of 174 GeV, which decays into two lighter Higgs bosons of 80 GeV. The intermediate light Higgs boson decays into two b quarks. Each test set consists of 10,000 samples that are pr-refined to the range of [550, 650] GeV. All the jet samples generated with pipeline of physics simulators.
sets, employ
Event Generation QCD jets extracted from QCD di-jet events that generated with MadGraph 4 for LHC TeV, followed by Pythia8 [6]] and Delphes 18 for parton shower and fast detector simulation. All jets are clustered using the anti-kr algorithm [9] with cone size R = 1.0 and a selection cut the jet transverse momentum pr > 450 GeV. We use the particle flow objects for jet clustering.
Input Preprocessing Jets are preprocessed before being fed into the neural models. Jets are longitudinally boosted and centered at (0, 0) in the (n, 6) plane. The centered jets are then rotated so that the jet principal axis ( n. F.
Σ. E. R, (with R. = f + of) and E, the constituent energy) is vertically aligned on the
(17 +) plane.
Hlype-pparametes Hyper- parameters are recorded Table chosen according to a jet classification task. We scan over the following hyper-parameter
parameter ranges: the transformer are
step size E [0.01, 0.1, 1.0, 10)
steps E (5, 10, 24, 60)
1r E le-3, le-4)
We found that fewer steps (5, 10] make training unstable. Thus model prefers relatively larger number of steps and a smaller step size. However, 60 steps MCMC takes much longer time to train and no significant improvement was observed for even longer chains. We balance training efficiency and effective learning by choosing 24 steps.
The MCMC chains are initialized with Gaussian noises, where the constituent features are sampled from the following distributions: log (22,, 1), m N'O0, 0.1%), and di NO, 0.2²).
Data
input features input length
(log(pr)) (); 2=1 N=40 with zero-padding
Energy Function
Number of layers NL
Model dimension dmoodel
128
Number of heads
Feed-forward dimension
1024
Dropout rate Normalization
0.
None
MCMC
Number of steps Step size Buffer size
0.1
10000
Resample rate Noise
0.05
0.005
Regularization
L2 Regularization
0.1
Training
Batch size Optimizer Learning rate
128
Adam (3) = 0.0, B2 = 0.999)
le-4 (decay rate 7 0.98)
Table 4: Model settings.
14

---

B Additional Results
B.1 Ablation Study
We explore the most erucial aspects of the model to test their functionality:
With an energy function approximated with a Multi-layer- (MLP) net, we were not able to achieve quality generation.
We also tried the Hamiltonian Monte Carlo (HMC) [22 for the MCMC procedure. However, we were not able achieve good performance these experiments.
Results measured in the lensen-Shannon Divergence of the high-level observables (pr and M) are recorded in Table 5
Ablation
JSD (pr) 10-4
JSD (M)/10)
Energy Function
MLP
Fig.
Fig.
MCMC
Dynamics
HMC
30
Table 5: Ablation study on different components of the model, training strategies, and training techniques. Since the MLP-based model is not able to produce high-quality samples, we instead show the observable distributions (Fig. 5
to visually show the failure patterns.
0.040
Data
Data
Model
Model
0.035
0.030
0.004
0.025
auu
0.003
0.020
0.015
0.002
0.010
0.001
0.005
0.000
0.000
200
400
600
800
1000
100
200
300
400
Pr/GeV
M/GeV
Figure 5: Typical high-level observable distributions for ML.P-bsed models.
B.2
Classification Performance of EBM-CLF
The classification accuracies for EBM BBM--LFF (QCD/W/Top 3-way classification) are recorded in Table EBM- CLF performs par with the fully supervised classifier ParticleNet (59), while FBM-CI. is trained much smaller dataset. The corresponding confusion matrices displayed Fig. 6 When empirically down-weight the term log p(y|x)) (decrease x in Eq. 7 the classification performance drops correspondingly.
log p(x, 9) = -log((x) + log p(y|x).
B.3
Additional Plots
In Fig. 7 we show the generated jet samples displayed in images on the (v,, Φ) plane. In Fig. we show the high-level observable distributions of generated jet samples of EBM-CLE In Fig. 9 we show the background distributions under different acceptance rates after cutting the energy score of the standard EBM.
15

---

Model
Top-1 1 Accuracy
Top-2 Accuracy
ParticleNet[59
0,871
0.976
EBM-CLF (K L.0)
0,850
0,969
EBM-CLF (A = 0.5)
0.708
0,906
EBM-CLF (k = 0.1)
0.679
0.852
Table Classification performance of HBM-CLA CLF on QCD/W/Top 3-way classification. & denotes the weight of the discriminative log-likelihood
0.5
0.81
0.07
0.11
0.84
0.05
0.10
0.7
0.5
Acbal
0.89
0.04
0.5
Actoal
0.90
0.04
0.4
-02
0.06
-0.2
Predicted
Predicted
Figure 6: Confusion matrices for EBM (BM-CLF(lef)) and ParticleNet(rrghn)).
021
115
65
k
Figure 7: Jet images averaged over 10000 jet samples. From left to right, show the initial random noises (left-most), EBM-generated jet samples by the MCMC chains intervals (middle), and Real jets (right-most)
C Reproducibility Statement
To ensure reproducibility and encourage open- practice in the HEP community, release the code implementation in /tttps://github. www./oollccenggee...ooo Beangg/BM-HHHH HEP The training sets and test sets accessible 47 14]. Due difficulties aligning model comparison protocols for different research groups, we thus focus on methods with code publicly available, that serve as credible baselines, model comparison.
16

---

Data
Generation
0.014
0.010
0.012
0.008
0.010
0.006
0.008
0.006
0.004
0.004
0.002
0.002
0.000
0.000
400
600
100
200
Pr/GeV
M/GeV
Figure 8: High-level observable distributions for the generated samples of EBM-CLF (orange) and the data (blue).
OCD 1.00
10-2
QCD Lc= 0.5]
QCD [<= 0.2]
QCD [e - 0.1]
a.u.
10-4
10-5
150
300
M/GeV
Figure 9: Background mass distributions under different acceptance rates e after cutting on the energy score from the EBM.
17


Reviews:
[{"review_id": "kHKIxYnpoG", "reviewer": "Reviewer_npbJ", "summary": "This paper explores the use of energy based models to modeling the distribution of jets in high energy particle physics. Here, the data consists of a vector of events, each of which is described by a momentum transverse to the beam and 2D spatial coordinates. The goal is to learn the true distribution of jets from real data via an energy based model, which can subsequently be used for Out-Of-Distribution detection and classification, in the manner of the paper \"Your classifier is secretly an energy based model and you should treat it like one\".\n\nThe model is trained with gradient ascent, using Langevin sampling to estimate the intractable expectation in the loss function, and making use of the (persistent) Contrastive Divergence of the loss. An additional KL term to minimize the difference between the true model distribution and the Langevin approximation is used. ", "questions": "Which are the original contributions of the paper on the machine learning side, if any? In particular, is the particular version of Contrastive Divergence used, (in the section KL Divergence-Improved EBM Training) key to your success here? I'd like to see more discussion of this KL term, which is quite interesting. \n\nI would be interested to know how this paper fits into a larger research project. Are there further directions that could be pursued, in terms of applications? Did the authors find the approach satisfactory, or were there limitations that concerned them? \n\nSuggestions:\n\nI was able to understand the structure of the paper fully only after reading the following two papers: https://arxiv.org/pdf/1912.03263.pdf and https://arxiv.org/pdf/1903.08689.pdf. The first describes the three applications of energy models you are interested in (generative modeling, OOD detection, and classification). The second describes the Contrastive divergence approach, and cites detailed notes which explain its derivation; your training algorithm appears to be a modification of their Algorithm 1. I would summarize your paper as \"an application of the approach of these papers to HEP Jet modeling\". While you do cite both of these papers, I would recommend highlighting them more, since they form the backbone of your approach and will help the reader to follow along.", "limitations": "Are there ways of incorporating domain specific models of HEP into the current approach? Currently it seems that the physics incorporated into simulators is not leveraged here. It would also be interesting if known expressions or heuristics for the energy could be incorporated into the energy function.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper appears to achieve success in a difficult domain of application. Their validation suggests that the energy based model they train successfully captures the desired data distribution, at least up to well-known features. Furthermore, they show that the energy based model can be altered to a supervised setting, and used as a classifier. \n\t\t\nThe use of the transformer as the neural architecture to model inter-particle interactions also appears to be a successful choice.", "weaknesses": "The ideas in this paper were often conveyed poorly, and it was quite hard to understand the technical details, either of the energy based model training procedure, or of the scientific problem in question. \n\nIn equation 2, x+ and x- were not clearly explained, making the paper hard to read for someone unfamiliar with the details of contrastive divergence. Using the expectation notation of (in Latex) $E_{x^+ \\sim p_D}$ would have helped clarify that the x+ are drawn from the data distribution and the x- from the (approximation of) the model distribution.\n\nAlgorithm 1 describes the \"stopping gradient\", but more information is needed here.\n\nThe equation in Figure 1 are blurry - it needs to be redone in higher resolution.\n\nMore explanation is needed for the anomaly detection task. I did not understand the set up in adequate detail. You say: \"With an energy-based model, which is trained on the QCD background events or directly on the slightly signal-contaminated data, we expect unseen signals to have higher energies and correspondingly lower likelihoods\". But what is a signal vs a background event here? This needs to be clear for a non-HEP audience.\n\nWhat the labels are in the classification task is unclear.\n\nThe paper notes that \"A more intriguing property of EBMs is that spurious correlations can be better handled\", but it wasn't clear to me either what spurious correlations referred to here, or why EBMs would be better.\n\n\"More interestingly, compositionality can be easily incorporated within the framework of EBMs by simply summing up the energy functions \": while I understand what is meant here, the word \"compositionality\" means many things in different contexts, and is too vague. While other papers on EBMs have used \"compositionality\" in this sense, you should either cite them, or explain directly what you mean by it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer npbJ,\n\nThank you for taking the time to review our manuscript and providing thoughtful feedback. We have made revisions to clarify the technical details. The following is the correspondence to the specific questions/comments.\n\n----- \n\n***Weaknesses:***\n> 1. *The ideas in this paper were often conveyed poorly, [...]*\n\nThanks for the feedback. We try our best to make the article understandable for both machine learning practitioners and domain scientists. We have made several clarifications in the revised manuscript (as discussed below).\n\n> 2. *In equation 2, x+ and x- were not clearly explained, [...]*\n\nThanks for the suggestion. We have incorporated it in our revised manuscript. We have adjusted Eq. 2 as $E_{x^+ \\sim p_D(x)} [\\nabla_\\theta E_\\theta (x^+)] - E_{x^- \\sim p_\\theta (x)} [\\nabla_\\theta E_\\theta (x^-)]$. We also adjusted other equations accordingly to improve clarity.\n\n> 3. *Algorithm 1 describes the \"stopping gradient\", but more information is needed here.*\n\nWe have added an explanation in Line 147 which we quote here: $E_{q(\\textbf{x})} [E_{\\hat \\theta}(\\textbf{x})]$ ( $\\hat \\theta$ denotes stopping gradient for the energy function because the gradient of this extra KL term is only propagated through the MCMC distribution $q_\\theta(\\textbf{x})$.)\n\n> 4. *The equation in Figure 1 are blurry - it needs to be redone in higher resolution.*\n\nThanks for the feedback. The image quality of Figure 1 is already improved. Please refer to the pdf attached to the global rebuttal for the improved version.\n\n> 5. *More explanation is needed for the anomaly detection task. [...]*\n\nAs indicated in the sentence, the background events are QCD jets, any other non-QCD jets can be considered signals. We clarified this point in the revised manuscript as follows:\n> With an energy-based model, [...], we expect unseen signals (i.e., non-QCD jets) to have higher energies and correspondingly lower likelihoods.\n\n> 6. *What the labels are in the classification task is unclear.*\n\nThe labels are jet types in the classification task as we are trying to classify different Standard Model jet types (QCD, W, and Top). Clarification is made in the revised manuscript.\n\n> 7. * [...] it wasn't clear to me either what spurious correlations referred to here, or why EBMs would be better.*\n\nPlease refer to section 4.2 (Line 163) for more details.\n\n> 8. *While other papers on EBMs have used \"compositionality\" in this sense, you should either cite them, or explain directly what you mean by it.*\n\nCitations were already present in the manuscript (at the end of this sentence).\n\n***Questions:***\n> 1. Which are the original contributions of the paper on the machine learning side, if any? [...]\n\nThere is no one single element that could serve as the magic key. The neural architecture in use, the training strategies, the hyper-parameters, the validation methods, and the test-time generation strategy are all crucial for the success. For instance, we used fewer Langevin Dynamics (LD) steps in the training iterations (24) and more steps in validation iterations (128) and test-time sampling (200), and we annealed the step size in test-time sampling.\n\nIn our experiments, the extra KL term is helpful in improving generation quality and training stability. The theoretical consideration largely follows that of [1]. Empirically, we found it most helpful to pass through all the LD steps (to improve stability and quality) and simply drop the entropy term (to accelerate training). \n\n> 2. I would be interested to know how this paper fits into a larger research project. [...]\n\nWe already presented a few most important applications for the Large Hadron Collider Physics. As stated in the *Introduction*, event generation and OOD detection are among the most crucial tasks in LHC physics. \n\nOne important motivation for this work is that the EBM provides us with a flexible framework to incorporate many complex tasks. It paves for a multi-purpose deep model that is adaptive, multi-functioning, performant, and robust. For instance, we can further build an EBM-based event generator that is controllable and facilitates generation prompts.\n\nThere is one limitation we consider important is the training and generation speed, since there are MCMC steps involved in each training iteration. We were able to balance training quality and speed by using short-run MCMC (24 LD steps) in training iterations. We will be motivated to further explore solutions to accelerate test-time generation.\n\n> 3. Suggestions:\nI was able to understand the structure of the paper fully only after reading the following [...] \n\nThanks for the suggestion. As summarized in previous responses, this work is a holistic approach to integrating important machine learning techniques in a scientific domain. There are quite a few references we consider important, and the two you mentioned are indeed very helpful. We reworked relevant parts of the manuscript to improve clarity and help the readers to trace back literature efficiently.\n\n***Limitations:***\n> Are there ways of incorporating domain specific models of HEP into the current approach? [...]\n\nThere are advantages that we take a bottom-up approach to directly learn from data about the underlying radiation patterns. However, it is possible to incorporate more physics inductive biases such as designing Lorentz-equivariant [2] energy models (these models are, however, computation-demanding and could slow down the EBM training even further). But that could be an interesting future research direction.\n\nIt is an interesting question of connecting physics energy and the energy in the probabilistic model. Due to the length limitation of the response, please refer to our response to Reviewer Xiwj for more information. Thanks!\n\n[1] Du, et al. \"Improved contrastive divergence training of energy based models.\" (2020).\n\n[2] Bogatskiy, et al. \"Lorentz group equivariant neural network for particle physics.\" (2020).\n\nBest regards,\\\nAuthors\n\n"}, {"review_id": "x8k8dUIN3I", "reviewer": "Reviewer_PfZ9", "summary": "This work applies recent EBM techniques to model jet streams from the LHC. The work adopts a transformer architecture for the EBM to allow a permutation invariant representation that captures high order relations between particles. EBM models of LHC particles are learned using techniques derived from recent image EBM works. Samples generated from the model are shown to closely match statistics of the observed data, and OOD detection with the learned EBM is performed and shown to have superior performance compared to VAE methods.", "questions": "* In the view of the authors, what are the most important novel aspects of their work?", "limitations": "Limitations were not discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This paper is the first to explore EBM learning in the area of high-energy physics. It is exciting and interesting to see research into EBMs for new and challenging domains.\n* Experimental results provide convincing evidence of the efficacy of the learned model for generation and OOD.", "weaknesses": "* The paper lacks theoretical and methodological novelty. The work primarily focuses on applications of EBMs to a new domain.\n* It appears that the primary practical application of the paper is OOD detection. In this case, it might be more effective to use an EBM designed specifically for OOD, as in [a]. I am not sure use of a generative EBM is well-justified. Tailoring the methodology more towards OOD would increase the focus of the paper, which tends to wander somewhat.\n\n[a] https://arxiv.org/pdf/2010.03759.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PfZ9,\n\nThank you for taking to time to review our manuscript and providing thoughtful feedback.\n\n-----\n\n***Weaknesses:***\n> *The paper lacks theoretical and methodological novelty. The work primarily focuses on applications of EBMs to a new domain.*\n> *It appears that the primary practical application of the paper is OOD detection. In this case, it might be more effective to use an EBM designed specifically for OOD, as in [a]. I am not sure use of a generative EBM is well-justified. Tailoring the methodology more towards OOD would increase the focus of the paper, which tends to wander somewhat.*\n\nThanks for the feedback. In this work, we established an energy-based probabilistic modelling framework for High Energy Physics events. As we elaborated in the manuscript (in both the *Introduction* and the *Problem Statement*), the EBM for HEP is a multi-purpose framework established for High Energy Physics.\n\nActually, OOD detection is only one of the practical sides of the EBM for HEP. And one of the most interesting part of EBM for detection OOD is that the energy score is generally decorrelated with jet mass (see the paragraph starting from line 263 in section 4.2).\n\nWe appreciate the suggestion of \"tailoring the methodology more towards OOD would increase the focus of the paper\", unfortunately, we can not agree with that approach. That would entail another OOD detection focused work, which is not the main purpose of our work.\n\n***Questions:***\n> *In the view of the authors, what are the most important novel aspects of their work?*\n\nAs stated above, we established an energy-based multi-purpose learning framework for High Energy Physics. This work is a holistic approach to integrating important machine learning methods and techniques in a scientific domain. It solves the most important tasks for the Large Hadron Collider physics in a data-driven manner. \n\nIn addition, one important advantage of this work is that the designed EBM provides us with a flexible framework to incorporate many complex tasks. It paves for a multi-purpose deep model that is adaptive, multi-functioning, performant, and robust. For instance, we can build an EBM-based event generator that is controllable and facilitates generation prompts.\n\n***Limitations:***\n> Limitations were not discussed.\n\nWe briefly mentioned the limitation of MCMC sampling in footnote 1. There is one limitation we consider important is the training and generation speed, since there are MCMC steps involved in each training iteration. We were able to balance training quality and speed by using short-run MCMC (24 LD steps) in training iterations.  It is also worth exploring other solutions to accelerate test-time generation. We would be happy to extend the discussion in the revised manuscript or in the appendix.\n\n-----\n\nHope we have addressed all of your concerns. Please let us know if you have any further questions or comments. \n\nBest regards, \\\nAuthors"}, {"review_id": "zqtlu7euEE", "reviewer": "Reviewer_Xiwj", "summary": "This paper proposes to use EBMs to model the fundamental interactions between elementary particles. The paper first introduces a set of techniques to enable effective learning of energy functions across elementary particles, including the use of a classifier to model the conditional distributions. The paper then proposes as an architecture to capture permutation invariant interaction between particles. The paper proposes a set of metrics to measure the generation quality of EBMs as a substitution for maximum likelihood estimation and illustrate how EBMs can effectively detect anomalies between particles.", "questions": "See weaknesses ", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper tackles an interesting and timely problem and explore how generative models (EBMs) can be used in the scientific application of particle dynamics.\n- The paper is clearly written and experiments demonstrates the utility of EBMs at both modeling the behaviors of underlying particles and detecting spurious patterns\n- The introduction compelling introduces and motivates the problem studied by the paper\n- In comparison to many other generative models, EBMs are much less well studied,  and this paper represents an original and interesting application of EBMs to different use cases \n- The method section has a variety of interesting modifications and tricks over prior work (such as the removal of the entropy term) that are interesting to read", "weaknesses": "- It might be interesting to see the extent to which the energy function learned by EBM correlates to real physical energy.\n- It might also be interesting to structure/augment the energy function with ground truth physical energy functions \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Xiwj,\n\nThank you for taking to time to review our manuscript and providing thoughtful feedback. The following is our correspondence to the weaknesses and questions.\n\n-----\n\n***Weaknesses:***\n>*It might be interesting to see the extent to which the energy function learned by EBM correlates to real physical energy.\nIt might also be interesting to structure/augment the energy function with ground truth physical energy functions*\n\nThank you for bringing up this very interesting question. First, we would like to clarify that the probabilistic energy and the physics energy are not the same. Though in some application cases, they might coincide, for instance in the Restricted Boltzmann Machine describing an Ising model. However, in the current setting, the energy of the probabilistic model is more of a concept trying to describe the radiation patterns within jets. If we restrict all the training jets to a specific physics energy (say 600 GeV), there are still variations in the radiation patterns and jet substructures. These variations are what the energy in the EBM is trying to describe and encode.\n\nIn addition, in order not to cause confusion, we added a brief clarification in the revised manuscript.\n\n-----\n\nHope we have addressed all the concerns. Please let us know if you have any other questions or comments. We would be happy to have further discussion.\n\nBest regards,\\\nAuthors\n"}, {"review_id": "Id0zXw7PFL", "reviewer": "Reviewer_bfxx", "summary": "In this paper, the authors describe an energy-based generative framework for modeling the behavior of elementary particles. This framework can then be used to generate High Energy Physics events, similar to those at LHC, as well as be used for anomaly detection, specifically QCD jets.", "questions": "1. In the paper the line \"The results of generative sampling are shown in Appendix C\" refers to actually refer to Appendix B.2. Please correct this if it is indeed the case. Also, please describe more the dataset used (for example, does it have class imbalances?) and consider including some confusion matrix or supplementary methods to better compare EBM-CLF to ParticleNet\n2. Please include in figure 4 left, the curve for EMB-CLF(E(x)) AUC(Top).\n3. Table 2 is a bit hard to read because of the many shades of gray and the inclusion of the background QCD classes. Please reconsider how to show the comparison more clearly.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strength of the paper lies in the clarity of writing, technical detail, and interesting application of the proposed methodology.", "weaknesses": "The main weakness of the paper is that it is hard to gauge how the improvements of the methodology over the baselines would actually translate in improvements of event detection at LHC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bfxx,\n\nThank you for taking the time to review our manuscript and providing thoughtful feedback. We have made revisions according to the comments. The following is the correspondence to the specific questions/comments.\n\n-----\n\n***Weaknesses:***\n*The main weakness of the paper is that it is hard to gauge how the improvements of the methodology over \nthe baselines would actually translate in improvements of event detection at LHC.*\n\nFirst, one of the most important advantages of these machine learning assisted methods is that they facilitate model-independent signal detection. That means we have a paradigm shift from theoretical-model-driven methods to data-driven methods effectively. The traditional approach of designing theoretical models, predicting signals, and excluding models is limiting in the sense that they can not cover all possible signal detection channels. With deep learning models directly trained on background events, we are now able to detect unobserved signals despite the underlying theoretical model. Based on that, we are trying to build deep learning models that are robust and sensitive to different possible signals. That's why we think EBM is a very promising candidate, given that it can be trained on background events in an unsupervised manner and it displays beautiful mass decorrelation without any auxiliary tasks (Fig. 3).\n\n***Questions:***\n\n1.1 *In the paper the line \"The results of generative sampling are shown in Appendix C\" refers to actually refer to Appendix B.2.\nPlease correct this if it is indeed the case.*\n\nThank you for pointing this out. This mismatch actually originates from the revision we made in the appendices during the supplementary material preparation phase. If you look at the supplementary material (we also included the main body for better readability), you can find that the reference to the appendices is correct.\n\n1.2 *Also, please describe more the dataset used (for example, does it have class imbalances?) and consider including some confusion \nmatrix or supplementary methods to better compare EBM-CLF to ParticleNet*\n\nThe datasets are described in Appendix A. There is no class imbalance. We have further clarified a few details in the revised manuscript. We quote the relevant part here for reference:\n> For the hybrid model EBM-CLF, we train on 300,000 simulated Standard Model jets (100,000 QCD jets, 100,000 boosted jets originating from the W boson, and 100,000 boosted jets originating from the top quark).\n\nWe also included the confusion matrices for reference in the attached pdf in the global rebuttal.\n\n2. *Please include in figure 4 left, the curve for EMB-CLF(E(x)) AUC(Top).*\n\nFigure 4 (left) has been updated. Please refer to the attached pdf in the global rebuttal. \n\n3. *Table 2 is a bit hard to read because of the many shades of gray and the inclusion of the background QCD classes.\nPlease reconsider how to show the comparison more clearly.*\n\nThank you for the suggestion. We have adjusted Table 2 to increase readability. Please refer to the pdf attached to the global rebuttal. Let us know if it is clear enough.\n\n-----\n\nWe hope that we have addressed all the concerns raised. Please let us know if you have any further comments or questions.\n\nBest regards, \n\nAuthors"}]

Author rebuttal:
Dear reviewers and AC,

We thank all the reviewers for your time and helpful feedback to make this article a better version. 

In the present study, we endeavor to synthesize various prior initiatives, with the goal of progressing toward a robust, multi-purpose modeling framework tailored for High Energy Physics (HEP) events. By constructing an energy-based model, we facilitate the probabilistic modeling of low-level feature distribution directly. The successful training of these EBMs has enabled us to generate high-quality events. A noteworthy observation is that the EBMs can be employed for model-independent Out-Of-Distribution (OOD) detection, while simultaneously remaining unencumbered by spurious correlations. In addition, the hybrid modelling combining EBM and supervised classification is an example of how we unify different physics tasks (jet classification, anomaly detection, and generative modeling) in a single framework.

According to the reviewers' comments, we have revised the manuscript to improve clarity. In summary,
* We clarified some technical details including but not limited to
    * notations in equations: such as $E_{x^+ \sim p_D(x)} [\nabla_\theta E_\theta (x^+)] - E_{x^- \sim p_\theta (x)} [\nabla_\theta E_\theta (x^-)]$ in Eq. 2.
    * domain-specific explanations: further explained the anomaly detection procedure in the context of new physics searches.
    * experimental details: improved dataset description in Appendix A.
* We fixed a few presentation issues in figures and tables (you can find the revised version in the attached pdf).

We hope that we have adequately responded to all of your inquiries and concerns. Should you have any additional questions or comments, please do not hesitate to let us know. We remain open to further suggestions that may enhance the quality of the manuscript.

Thank you again for your time!

Best regards, \
Authors

Meta-review: I am thrilled to convey my recommendation for the acceptance of your paper to be presented at NeurIPS. 

This is an application of energy based models to LHC applications. It almost entirely lacks in novelty and would probably be rejected if not in the physical sciences track.  However, it is in the physical sciences track and the reviews and discussion seem sufficiently positive to put it over the line.

NeurIPS is a prestigious platform that attracts the brightest minds in the field, and your paper's acceptance adds to the conference's reputation for excellence. I am genuinely excited to see your work presented and discussed among peers who share your passion for pushing the boundaries of knowledge.
Decision: Accept (poster)