Title: Versatile Energy-Based Probabilistic Models for High Energy Physics
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative modeling

Energy-based models

Out-of-distribution detection

Sciences

Application

Physics
Abstract: As a classical generative modeling approach, energy-based models have the natural advantage of flexibility in the form of the energy function. Recently, energy-based models have achieved great success in modeling high-dimensional data in computer vision and natural language processing. In line with these advancements, we build a multi-purpose energy-based probabilistic model for High Energy Physics events at the Large Hadron Collider.  This framework builds on a powerful generative model and describes higher-order inter-particle interactions. It suits different encoding architectures and builds on implicit generation. As for applicative aspects, it can serve as a powerful parameterized event generator for physics simulation, a generic anomalous signal detector free from spurious correlations, and an augmented event classifier for particle identification.

Versatile Energy-Based Probabilistic Models for High Energy Physics

Abstract

As a classical generative modeling approach, energy-based models have the natural advantage of flexibility in the form of the energy function. Recently, energy-based models have achieved great success in modeling high-dimensional data in computer vision and natural language processing. In line with these advancements, we build a multi-purpose energy-based probabilistic model for High Energy Physics events at the Large Hadron Collider. This framework builds on a powerful generative model and describes higher-order inter-particle interactions. It suits different encoding architectures and builds on implicit generation. As for applicative aspects, it can serve as a powerful parameterized event generator for physics simulation, a generic anomalous signal detector free from spurious correlations, and an augmented event classifier for particle identification.

1 Introduction

The Large Hadron Collider (LHC) [1], the most energetic particle collider in human history, collides highly energetic protons to examine the underlying physics of subatomic particles and extend our current understanding of the fundamental forces of nature, summarized by the Standard Model of particle physics. After the great success in observing the Higgs boson [2, 28], the most critical task of searching for new physics signals remains challenging. High Energy Physics (HEP) events produced at the LHC have the properties of high dimensionality, high complexity, and enormous data size. To detect rare signals from enormous background events, physics observables describing these patterns have been used to identify different types of radiation patterns. However, it's not possible to utilize all the information recorded by the detectors with a few expert-designed features. Thus deep neural classifiers and generative models, which can easily process high-dimensional data meet the needs for more precise data analysis and signal detection.

Energy-based Models (EBMs) [38, 3, 46], as a classical generative framework, leverage the energy function for learning dependencies between input variables. With an energy function E ( x ) and constructing the un-normalized probabilities through the exponential ˜ p ( x ) = exp( -E ( x )) , the energy model naturally yields a probability distribution. Despite the flexibility in the modeling, the training of EBMs has been cumbersome and unstable due to the intractable partition function and the corresponding Monte Carlo sampling involved. More recently, EBMs have been succeeding in high-dimensional modeling [55, 56, 25, 23, 62, 19, 51] for computer vision and natural language processing. At the same time, it has been revealed that neural classifiers are naturally connected with EBMs [67, 33, 32], combining the discriminative and generative learning processes in a common learning regime. More interestingly, compositionality [30] can be easily incorporated within the framework of EBMs by simply summing up the energy functions [21, 22]. On the other hand, statistical physics originally inspired the invention of EBMs. This natural connection in formalism makes EBMs appealing in modeling physical systems. In physical sciences, EBMs have been used to

simulate condensed-matter systems and protein molecules [57]. They have also been shown great potential in structure biology [24], in a use case of protein conformation.

Motivated by the flexibility in the architecture and the compatibility with different tasks, we explore the potential of EBMs in modeling radiation patterns of elementary particles at high energy. The energy function is flexible enough to incorporate sophisticated architectures. Thus EBMs provide a convenient mechanism to simulate complex correlations in input features or high-order interactions between particles. Aside from image generation, applications for point cloud data [68], graph neural networks [48] for molecule generation are also explored. In particle physics, we leverage the self-attention mechanism [6, 65], to mimic the complex interactions between elementary particles.

As one important practical application, neural net-based unsupervised learning of physics events [58, 42, 8, 64] have been explored in the usual generative modeling methods including Generative Adversarial Networks (GANs) [31] and Variational Autoencoders (VAEs) [45]. However, GANs employ separate networks, which need to be carefully tuned, for the generation process. They usually suffer from unstable training, high computation demands, and mode collapse. In comparison, VAEs need a well-designed reconstruction loss, which could be difficult for sophisticated network architectures and complex input features. EBMs thus provide a strong alternative generative modeling framework of LHC events, by easily incorporating sophisticated physics-inspired neural net architectures.

At the same time, EBMs can serve as generic signal detectors, since out-of-distribution (OOD) detection comes naturally in the form of energy comparison. More importantly, EBMs incur fewer spurious correlations in OOD detection. This plays a slightly different role in the context of signal searches at the LHC. There are correlations that are real and useful but at the same time hinder effective signal detection. As we will see in Section 4.2, the EBMs are free from the notorious correlation observed in many anomaly detection methods in HEP, in both the generative and the discriminative approaches.

As summarized in Table 1, we build a multi-tasking framework for High Energy Physics. To that end, we construct an energy-based model of the fundamental interactions of elementary particles to simulate the resulting radiation patterns. We especially employ the short-run Markov Chain Monte Carlo for the EBM training. We show that EBMs are able to generate realistic event patterns and can be used as generic anomaly detectors free from spurious correlations. We also explore EBM-based hybrid modeling combining generative and discriminative models for HEP events. This unified learning framework paves for future-generation event simulators and automated new physics search strategies. It opens a door for combining different methods and components towards a powerful multi-tasking engine for scientific discovery at the Large Hadron Collider.

2 Problem Statement

Describing HEP Events Most particle interactions happening at the LHC are governed by Quantum Chromodynamics (QCD), due to the hadronic nature of the proton-proton collision. Thus jets are enormously produced by these interactions. A jet is formed by collimated radiations originating from highly energetic elementary particles (e.g., quarks, gluons, and sometimes highly-boosted electro-weak bosons). The tracks and energy deposits of the jets left in the particle detectors reveal the underlying physics in complex patterns. A jet may have hundreds of jet constituents, resulting in high-dimensional and complex radiation patterns and bringing difficulties in encoding all the information with expert features. Reconstructing, identifying, and classifying these elementary particles manifest in raw data is thus critical for ongoing physics analysis at the LHC.

Specifically, each constituent particle within a jet has (log p T , η, ϕ ) as the descriptive coordinates in the detector's reference frame. Here, p T represents the transverse momentum perpendicular to the

beam axis, and ( η, ϕ ) denotes the spatial coordinates within the cylindrical detector. (More details about the datasets can be found in Appendix A.) Thus a jet can be described by the high-dimensional vector x = { (log p T , η, ϕ ) i } N i , supposing the jet has N constituents. Our goal is thus to model the data distribution p ( x ) precisely, and p ( y | x ) in the case of supervised classification with y denoting the corresponding jet type.

Parameterized Generative Modeling At the LHC, event simulation serves as an important handle for background estimation and data analysis. For many years, physics event simulators [10] have been built on Monte Carlo methods based on physics rules. These generators are slow and need to be tuned to the data frequently. Deep neural networks provide us with an efficient parameterized generative approach to event simulation for the coming decades. Generative modeling in LHC physics has been experimented with GANs (image-based [58, 27] and point-cloud-based [43]) and VAEs [64]. There are also GANs [8] working on high-level features for event selection. However, as mentioned in the introduction, these models all require explicit generators, which brings obstacles to situations with complex data formats and sophisticated neural architectures.

OODDetection for New Physics Searches Despite the great efforts in searching for new physics signals at the LHC, there is no hint of beyond-Standard-Model physics. Given the large amount of data produced at the LHC, it has been increasingly challenging to cover all the possible search channels. Experiments at the LHC over the past decades have been focused on model-oriented searches, as in searching for the Higgs boson [26, 36]. The null results up to now from model-driven searches call for novel solutions. We thus shift to model-independent and data-driven anomaly detection strategies, which are data-oriented rather than theory-guided, for new physics searches.

Neural nets-based anomaly detection in HEP takes different formats. Especially, unsupervised generative models trained on background events can be used to detect potential unseen signals. More specifically, Variational Autoencoders [15, 41, 5] have been employed to detect novel signals, directly based on detector-level features. High-level observables-based V AEs [12, 39] have also been explored. However, naively trained VAEs (and Autoencoders) are usually subject to spurious correlations which result in failure modes in anomaly detection, partially due to the format of the reconstruction loss involved. Usually, one needs to explicitly mitigate these spurious correlations with auxiliary tasks such as outlier exposure [35, 15], or modified model components such as autoencoders with smeared pixel intensities [29] and autoencoders augmented with energy-based density estimation [69, 20].

At the same time, EBM naturally has the handle for discriminating between in-distribution and outof-distribution examples and accommodates tolerance for spurious correlations. While in-distribution data points are trained to have lower energy, energies of OOD examples are pushed up in the learning process. This property has been used for OOD detection in computer vision [25, 33, 23, 71]. This thus indicates great potential for EBM-based new physics detection at the LHC.

3 Methods

3.1 Energy Based Models

Energy-based models are constructed to model the unnormalized data probabilities. They leverage the property that any exponential term exp( -E ( x )) is non-negative and thus can represent unnormalized probabilities naturally. The data distribution is modeled through the Boltzmann distribution: p θ ( x ) = exp( -E θ ( x )) /Z ( θ ) with the energy model E θ ( x ) 1 : X → R mapping x ∈ X to a scalar. And the partition function Z = ∫ ˜ p ( x ) d x = ∫ exp( -E θ ( x )) d x integrates over all the possible states.

EBMs can be learned through maximum likelihood E p D (log p θ ( x )) . However, the training of EBMs can pose challenges due to the intractable partition function in log p θ ( x ) = -E θ ( x ) -log Z ( θ ) . Though the partition function is intractable, the gradients of the log-likelihood do not involve the partition function directly. Thus when taking gradients w.r.t. the model parameters θ , the partition

Multi-Head

Attention

Linear Embedding

Feed Forward

Add(&Norm)

Add(&Norm)

(batch_size, len_seq, 3)

MLP

E

Gradient-based

Contrastive

Divergence

MCMC

Samples

Data

function is canceled out. The gradient of the maximum likelihood loss function can be written as:

where p D ( x ) is the data distribution and p θ ( x ) is the model distribution. The training objective is thus composed of two terms, corresponding to two different learning phases (i.e., the positive phase to fit the data x + , and the negative phase to fit the model distribution x -). When parameterizing the energy function with feed-forward neural networks [54], the positive phase is straightforward. However, the negative phase requires sampling over the model distribution. This leads to various Monte Carlo sampling strategies for estimating the maximum likelihood.

Contrasting the energies of the data and the model samples as proposed Contrastive Divergence (CD) [37, 11] leads to an effective strategy to train EBMs with the following CD objective:

where T denotes the one-step Monte Carlo Markov Chain (MCMC) kernel imposed on the data distribution. In more recent approaches for high-dimensional modeling [56, 25], we can directly initialize from random noises to generate MCMC samples instead of initializing from the data distribution as in the original CD approach. This strategy also helps with exploring and mixing between modes.

Negative Sampling The most critical component in the negative phase is the sampling to estimate the model distribution. We employ gradient-based MCMC generation for the negative sampling, which is handled by Langevin Dynamics [52, 66]. As written in Eq. 4, Langevin dynamics uses gradients w.r.t. the input dimensions, associated with a diffusion term to inject stochasticity, to generate a sequence of negative samples { x -k } K k =1 .

MCConvergence The training anatomy [55, 56] for short-run non-convergent MCMC and long-run convergent MCMC shows that short-run ( ∼ 5-100 steps) MCMC initialized from random distributions

is able to generate realistic samples, while long-run MCMC might be oversaturated with lower-energy states.

To improve mode coverage, we use random noise to initialize MCMC chains. To accelerate training, we employ a relatively small number of MCMC steps. In practice, we can reuse the generated samples as initial samples of the following MCMC chains to accelerate mixing, similar to Persistent Contrastive Divergence [63]. Following the procedure in [25], we use a randomly initialized buffer that is consistently updated from previous MCMC runs as the initial samples. (As empirically shown [13, 66], a Metropolis-Hastings step is not necessary. So we ignore this rejection update in our experiments.)

Energy Function Since there is no explicit generator in EBMs, we have much freedom in designing the architectures of the energy function. This also connects with the fast-paced development of supervised neural classifier architectures for particle physics. We can directly reuse the architectures from supervised classifiers in the generative modeling of EBMs. We use a self-attention-based transformer to parameterize the energy function E θ ( · ) . We defer the detailed description to Sec. 3.3.

The full algorithm for training EBMs is described in Algorithm 1.

Algorithm 1 EBM training with MCMC by Langevin Dynamics

Input: training samples { x + i } N i =1 from p D ( x ) , parameterized energy function E θ ( · ) , initial buffer B ← ∅ , Langevin dynamics step size λ x , number of MCMC steps K, model parameter learning rate λ θ , regularization strength α

for Gradient descent step l = 0...L-1 do x -i, 0 ∼ 0 . 95 ∗ B +0 . 05 ∗ U ▷ Reinitialize the samples in the buffer with random noise in the probability of 0.05

for Langevin dynamics step k = 0...K-1 do x -i,k +1 = x -i,k -λ x ∇ x E θ ( x -i,k ) + 0 . 005 · ϵ k , ϵ k ∼ N (0 , 1) ▷ Langevin Dynamics taking gradients w.r.t. input dimensions

end for

3.2 Hybrid Modeling

Neural Classifier as an EBM A classical classifier can be re-interpreted in the framework of EBMs [46, 33, 67, 17, 40], with the logits g ( x ) corresponding to negative energies of the joint distribution p ( x , y ) = exp( g ( x ) y ) Z , where g ( x ) y denotes the logit corresponding to the label y. Thus the probability marginalized over y can be written as p ( x ) = ∑ y exp( g ( x ) y ) Z , with the energy of x as -log ∑ y exp( g ( x ) y ) . We are then brought back to the classical softmax probability exp( g ( x ) y ) ∑ y exp( g ( x ) y ) when calculating p ( y | x ) .

This viewpoint provides a novel method for jointly training a supervised classifier and an unsupervised generative model. Specifically, [33] successfully incorporated EBM-based generative modeling into a classifier in image generation and classification. We follow their proposal to train the hybrid model as follows to ensure the classifier p ( y | x ) is unbiased. The joint log-likelihood is decomposed into two terms:

Thus one can maximize log p ( x ) with the contrastive divergence of the EBM with the energy function E ( x ) = -log ∑ y exp( g ( x ) y ) , and maximize log p ( y | x ) with the usual cross-entropy of the classification.

▷ L 2 Regularization ▷ Update model parameters with gradient descent ▷ Update the buffer with generated samples

3.3 Energy-based Models for Elementary Particles

We would like to construct an energy-based model for describing jets and their inner structures. In conceiving the energy function for these elementary particles, we consider the following constraints and characteristics: 1) permutation invariance - the energy function should be invariant to jet constituent permutations, and 2) higher-order interactions - we would like the energy function to be powerful enough to simulate the complex inter-particle interactions.

Thus, we leverage the self-attention-based transformer [65] to approximate the energy function, which takes into account the higher-order interactions between the component particles. As indicated in Eq. 6b, the encoding vector of each constituent W is connected with all other constituents through the self-attention weights A in Eq. 6a, which are already products of particle representations Q and K .

Moreover, we can easily incorporate particle permutation invariance [70] in the transformer, by summing up the encodings of each jet constituent. The transformer architecture is shown in Fig. 1. The coordinates (log p T , η, ϕ ) of each jet constituent are first embedded into a d model -dimensional space through a linear layer, then fed into N L self-attention blocks sequentially. After that, a sumpooling layer is used to sum up the features of the jet constituents. Finally, a multi-layer-perceptron projector maps the features into the energy score. Model parameters are recorded in Table 4 of Appendix A.

4 Experiments

Training Details The training set consists of 300,000 QCD jets. We have 10,000 samples in the buffer and reinitialize the random samples with a probability of 0.05 in each iteration. We use a relatively small number of steps (e.g., 24) for the MCMC chains. The step size λ x is set to 0.1 according to standard deviations of the input features. The diffusion magnitude within the Langevin dynamics is set to 0.005. The number of steps used in validation steps is set to 128 for better mixing.

We use Adam [44] for optimization, with the momenta β 1 = 0 . 0 and β 2 = 0 . 999 . The initial learning rate is set to 1e-4, with a decay rate of 0.98 for each epoch. We use a batch size of 128, and train the model for 50 epochs. More details can be found in Appendix A.

Model Validation In monitoring the likelihood, the partition function can be estimated with Annealed Importance Sampling (AIS) [53]. However, these estimates can be erroneous and consume a lot of computing resources. Fortunately for physics events, we have well-designed high-level features as a handle for monitoring the generation quality. Especially, we employ the boost-invariant jet transverse momentum p T = ∑ N i =1 p Ti and the Lorentz-invariant jet mass M = √ ( ∑ N i =1 E i ) 2 -( ∑ N i =1 p i ) 2 as the validation observables. And we calculate the Jensen-Shannon Divergence (JSD), between these high-level observable distributions of the data and the model generation, as the metric. In contrast to the short-run MCMC in the training steps, we instead use longer MCMC chains for generating the validation samples.

When we focus on downstream tasks such as OOD detection, it's reasonable to employ the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) as the validation metric, with Standard Model top jets serving as the benchmark signal.

Generation Test-time generation is achieved in MCMC transition steps from the proposal random (Gaussian) distribution. We use a colder model and a smaller step size at test time which is annealed from 0.1 and decreases by a factor of 0.8 in every 40 steps to encourage stable generation. The diffusion noise magnitude is set to 0.001. Longer MCMC chains with more steps (e.g., 200) are taken to achieve realistic generation. 2 A consistent observation across various considered methods is that the step size stands out as the crucial parameter predominantly influencing the quality of generation.

Anomaly Detection In contrast to prevalent practices in computer vision, EBM-based OOD detection in LHC physics exhibits specificity. The straightforward approach of comparing two distinct datasets, such as CIFAR10 and SVHN, overlooks the intricate real-world application environments. Adapting OOD detection to scientific discovery at the LHC, we reformulate and tailor the decision process as follows: if we train on multiple known Standard Model jet classes, we focus on classconditional model evaluation for discriminating between the unseen test signals and the most copious background events (i.e., QCD jets rather than all the Standard Model jets).

4.1 Generative Modeling - Energy-based Event Generator

We present the generated jets transformed from initial random noises with the Langevin dynamics MCMC. Due to the non-human-readable nature of physics events (e.g., low-level raw records at the particle detectors), we are not able to examine the generation quality through formats such as images directly. However, it has a long history that expert-designed high-level observables can serve as strong discriminating features. In the first row of Fig. 2, we first show the distributions of input features for the data and the model generation. Meanwhile, in the second row, we plot the distributions of high-level expert observables including the jet transverse momentum p T and the jet mass M. Through modeling low-level features in the detector space, we achieve precise recovery of the high-level physics observables in the theoretical framework. For better visualization, we can also map the jets onto the ( η, ϕ ) plane, with pixel intensities associated with the corresponding energy deposits. We show the average generated jet images in Fig. 7 of Appendix B, comparing to the real jet images ( right-most ) in the ( η, ϕ ) plane. In Table 2, we present the Jensen-Shannon Divergence of the high-level observables p T and M distributions between real data and model generation, as the quantitative measure of the generation performance.

2

0

2

4

6

log

pTi

/GeV

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Data

Model Generation

1.0

0.5

0.0

0.5

1.0

0

1

2

3

4

5

Data

Model Generation

1.0

0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Data

Model Generation

200

300

400

500

600

700

800

900

1000

pT

/GeV

0.000

0.001

0.002

0.003

0.004

0.005

0.006

0.007

Data

Model Generation

0

100

200

300

400

500

M/GeV

0.000

0.002

0.004

0.006

0.008

0.010

0.012

Data

Model Generation

0.0

0.2

0.4

0.6

0.8

1.0

M/

pT

0

1

2

3

4

5

6

Data

Model Generation

4.2 Anomaly Detection - Anomalous Jet Tagging

Since EBMs naturally provide an energy score for each jet, for which the in-distribution samples should have lower scores while OOD samples are expected to incur higher energies. Furthermore, a classifier, when interpreted as an energy-based model, the transformed energy score can also serve as an OOD identifier [33, 49].

EBM In HEP, the in-situ energy score can be used to identify potential new physics signals. With an energy-based model, which is trained on the QCD background events or directly on the slightly signal-contaminated data, we expect unseen signals (i.e., non-QCD jets) to have higher energies and correspondingly lower likelihoods.

In Fig. 3, we compare the energy distributions of in-distribution QCD samples, out-of-distribution signal examples (hypothesized heavy Higgs boson [7] which decays into four QCD sub-jets), and random samples drawn from the proposal Gaussian distribution. We observe that random samples unusually have the highest energies. Signal jets have relatively higher energies compared with the QCD background jets, making model-independent new physics searches possible.

Spurious Correlation A more intriguing property of EBMs is that spurious correlations can be better handled. Spurious correlations in jet tagging might result in distorted background distributions and obscure effective signal detection. For instance, V AEs in OOD detection can be highly correlated with the jet masses [15], similar to the spurious correlation with image pixel numbers in computer vision [50, 60]. In the right panel of Fig. 3, we plot the correlation between the energy score and the jet mass. Unlike other generative strategies for model-independent anomaly detection, EBMs are largely free from the spurious correlation between the energy E ( x ) and the jet mass M. The underlying reason for EBMs not presenting mass correlation could be the negative sampling involved in the training process. Larger mass modes are already covered during the negative sampling process. This makes EBMs a promising candidate for model-independent new physics searches.

2

1

0

1

2

E

0

2

4

6

8

10

QCD

Signal

Random

0

100

200

300

400

MJ

/GeV

2

1

0

1

2

E

QCD

Signal

Random

EBM-CLF The task of classifying/identifying different Standard Model jet types and the task of searching for beyond the Standard Model signals actually can be unified in a single approach with neural classifiers distinguishing different Standard Model particle types [16]. Compared with naive generative model based OOD, training neural classifiers on different known Standard Model jets helps the model with learning more meaningful and robust representations for effectively detecting new signals. Further combining generative models and discriminative classifiers can be achieved in the EBM framework elegantly. We employ the hybrid learning scheme [54, 33] combining the discriminate and the generative approaches. It links with many OOD detection techniques and observations. For instance, the maximum logit for OOD detection [34] comes in the format of the lowest energy min y E ( x , y ) in this framework.

We train an EBM-based multi-class classifier (EBM-CLF) according to Eq. 5, for both discriminating different Standard Model jets (QCD jets, boosted W jets, and boosted top jets) and generating real-like jet samples. The results of generative sampling are shown in Appendix B. The jointly trained EBM and jet classifier maintain the classification accuracy (see Appendix B). The associated EBM is augmented by the discriminative task, and thus assisted with better inductive biases and domain knowledge contained in the in-distribution classes. EBM-CLF is an example of how we unify different physics tasks (jet classification, anomaly detection, and generative modeling) in a unified framework.

We measure the OOD detection performance in the ROC curve and the AUC of the binary classification between the background QCD samples and the signal jets. Table 3 records the AUCs of different models (and different anomaly scoring functions) in tagging Standard Model Top jets and hypothesized Higgs bosons (OOD H ). The jointly trained model generally has better anomaly tagging performance compared with the naive EBM. We also explore the norm of the score function of p θ ( x ) : ∥∇ x log p θ ( x ) ∥ = ∥∇ x E ( x ) ∥ serving as the anomaly score (similar to the approximate mass in [33]). Constructed from the energy function, the score function approximately inherits the nice property of mass decorrelation. However, they have slightly worse performance compared with E ( x ) . The corresponding ROC curves are shown in the left panel of Fig. 4, in terms of the signal efficiency ϵ S (i.e., true positive rate ) and the background rejection rate 1 /ϵ B (i.e., 1/ false positive rate ). In the right panel, we plot the background mass distributions under different cuts on the energy scores. We observe excellent jet mass decorrelation/invariance for energy score-based anomalous jet tagging. Additionally for EBM-CLF, we also record the AUCs for anomaly scores of the class-conditional softmax probability p ( y | x ) and the logit g ( x ) y = -E ( x , y ) corresponding to the background QCD class, in Table 3. However, without further decorrelation strategies (auxiliary tasks or assistant training strategies), these two anomaly scores are usually strongly correlated with the masses of the in-distribution classes and distort the background distributions. Thus we list the results here only for reference.

5 Conclusion

We present a novel energy-based generative framework for modeling the behavior of elementary particles. By mimicking the inter-particle interactions with a self-attention-based transformer, we map the correlations in the detector space to a probabilistic space with an energy function. The energy model is used for the implicit generation of physics events. Despite the difficulty in training EBMs, we adapted the training strategies to balance learning efficiency and training stability. We adopt short-run MCMC sampling at training, while at test time we instead use dynamic generation to obtain prominent generative quality. Additionally, the framework supports the construction of an augmented classifier with integrated generative modeling. This unified approach provides us with flexible tools for high-performance parameterized physics event simulation and spurious-correlationfree model-independent signal detection. It marks a significant stride in next-generation multitasking machine learning models for high-energy physics.

0.2

0.4

0.6

0.8

1.0

10 0

10 1

10 2

1/

B

random

EBM(E(x)), AUC(Top)=0.679

EBM-CLF(E(x)), AUC(Top)=0.703

EBM-CLF(E(x)), AUC(H)=0.815

0

50

100

150

200

250

300

350

MJ

/GeV

10

5

10

4

10

3

10

2

QCD [

=1.0

]

QCD [

=0.5

]

QCD [

=0.2

]

QCD [

=0.1

]

S

Acknowledgments and Disclosure of Funding

This work is partially supported by the IV ADO Postdoctoral Research Funding. We acknowledge the computing resources provided by Mila (mila.quebec).

References

LHC Machine. JINST , 3:S08001, 2008.

Georges Aad et al. Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B , 716:1-29, 2012.

David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science , 9(1):147-169, 1985.

Johan Alwall, Michel Herquet, Fabio Maltoni, Olivier Mattelaer, and Tim Stelzer. Madgraph 5: going beyond. Journal of High Energy Physics , 2011(6), Jun 2011.

Oliver Atkinson, Akanksha Bhardwaj, Christoph Englert, Partha Konar, Vishal S. Ngairangbam, and Michael Spannowsky. IRC-Safe Graph Autoencoder for Unsupervised Anomaly Detection. Front. Artif. Intell. , 5:943135, 2022.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR , abs/1409.0473, 2015.

G.C. Branco, P.M. Ferreira, L. Lavoura, M.N. Rebelo, Marc Sher, and Joao P. Silva. Theory and phenomenology of two-Higgs-doublet models. Phys. Rept. , 516:1-102, 2012.

Anja Butter, Tilman Plehn, and Ramon Winterhalder. How to GAN LHC Events. SciPost Phys. , 7(6):075, 2019.

Matteo Cacciari, Gavin P Salam, and Gregory Soyez. The anti-kt jet clustering algorithm. Journal of High Energy Physics , 2008(04):063-063, Apr 2008.

J. M. Campbell et al. Event Generators for High-Energy Physics Experiments. In 2022 Snowmass Summer Study , 3 2022.

Miguel Á. Carreira-Perpiñán and Geoffrey Hinton. On contrastive divergence learning. In Robert G. Cowell and Zoubin Ghahramani, editors, Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics , volume R5 of Proceedings of Machine Learning Research , pages 33-40. PMLR, 06-08 Jan 2005. Reissued by PMLR on 30 March 2021.

Olmo Cerri, Thong Q. Nguyen, Maurizio Pierini, Maria Spiropulu, and Jean-Roch Vlimant. Variational Autoencoders for New Physics Mining at the Large Hadron Collider. JHEP , 05:036, 2019.

Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference on Machine Learning , 2014.

Taoli Cheng. Test sets for jet anomaly detection at the lhc, March 2021.

Taoli Cheng, Jean-François Arguin, Julien Leissner-Martin, Jacinthe Pilette, and Tobias Golling. Variational autoencoders for anomalous jet tagging. Phys. Rev. D , 107(1):016002, 2023.

Taoli Cheng and Aaron Courville. Invariant representation driven neural classifier for anti-QCD jet tagging. JHEP , 10:152, 2022.

Jifeng Dai and Ying Nian Wu. Generative modeling of convolutional neural networks. CoRR , abs/1412.6296, 2014.

J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi. Delphes 3: a modular framework for fast simulation of a generic collider experiment. Journal of High Energy Physics , 2014(2), Feb 2014.

Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc'Aurelio Ranzato. Residual energy-based models for text generation. In International Conference on Learning Representations , 2020.

Barry M. Dillon, Luigi Favaro, Tilman Plehn, Peter Sorrenson, and Michael Krämer. A normalized autoencoder for lhc triggers, 2022.

Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models.

Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In NeurIPS , 2021.

Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models.

Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomicresolution protein conformations. In International Conference on Learning Representations , 2020.

Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models.

F. Englert and R. Brout. Broken symmetry and the mass of gauge vector mesons. Phys. Rev. Lett. , 13:321-323, Aug 1964.

Martin Erdmann, Jonas Glombitza, and Thorben Quast. Precise simulation of electromagnetic calorimeter showers using a Wasserstein Generative Adversarial Network. Comput. Softw. Big Sci. , 3(1):4, 2019.

S. Chatrchyan et al. Observation of a new boson at a mass of 125 gev with the cms experiment at the lhc. Physics Letters B , 716(1):30-61, 2012.

Thorben Finke, Michael Krämer, Alessandro Morandini, Alexander Mück, and Ivan Oleksiyuk. Autoencoders for unsupervised anomaly detection in high energy physics. JHEP , 06:161, 2021.

J.A. Fodor and E. LePore. The Compositionality Papers . Clarendon Press, 2002.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020.

Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Kristjanson Duvenaud. No mcmc for me: Amortized sampling for fast and stable training of energy-based models. ArXiv , abs/2010.04230, 2021.

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one.

Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Xiaodong Song. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning , 2022.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations , 2019.

Peter W. Higgs. Broken symmetries, massless particles and gauge fields. Phys. Lett. , 12:132-133, 1964.

Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput. , 14(8):1771-1800, aug 2002.

J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences , 79(8):2554-2558, 1982.

Pratik Jawahar, Thea Aarrestad, Nadezda Chernyavskaya, Maurizio Pierini, Kinga A. Wozniak, Jennifer Ngadiuba, Javier Duarte, and Steven Tsan. Improving Variational Autoencoders for New Physics Detection at the LHC With Normalizing Flows. Front. Big Data , 5:803685, 2022.

Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Alan Kahn, Julia Gonski, Inês Ochoa, Daniel Williams, and Gustaaf Brooijmans. Anomalous jet identification via sequence modeling. JINST , 16(08):P08012, 2021.

Raghav Kansal, Javier Duarte, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopulos. Graph Generative Adversarial Networks for Sparse Data Generation in High Energy Physics. In 34th Conference on Neural Information Processing Systems , 11 2020.

Raghav Kansal, Javier Duarte, Hao Su, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopulos. Particle Cloud Generation with Message Passing Generative Adversarial Networks. 6 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR , abs/1312.6114, 2014.

Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. 2006.

Julien Leissner-Martin, Taoli Cheng, and Jean-Francois Arguin. Qcd jet samples with particle flow constituents, July 2020.

M. Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with energy-based models. ArXiv , abs/2102.00546, 2021.

Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? ArXiv , abs/1810.09136, 2019.

Subhajit Naskar, Pedram Rooshenas, Simeng Sun, Mohit Iyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. In ACL , 2021.

Radford Neal. MCMC Using Hamiltonian Dynamics. In Handbook of Markov Chain Monte Carlo , pages 113-162. 2011.

Radford M. Neal. Annealed importance sampling. Statistics and Computing , 11:125-139, 2001.

Jiquan Ngiam, Zhenghao Chen, Pang Wei Koh, and Andrew Y. Ng. Learning deep energy models. In Proceedings of the 28th International Conference on International Conference on Machine Learning , ICML'11, page 1105-1112, Madison, WI, USA, 2011. Omnipress.

Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In AAAI , 2020.

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. In NeurIPS , 2019.

Frank Noé, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science , 365, 2019.

Michela Paganini, Luke de Oliveira, and Benjamin Nachman. Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters. Phys. Rev. Lett. , 120(4):042003, 2018.

Huilin Qu and Loukas Gouskos. ParticleNet: Jet Tagging via Particle Clouds. Phys. Rev. D , 101(5):056019, 2020.

J. Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. ArXiv , abs/1906.02845, 2019.

Torbjörn Sjöstrand, Stephen Mrenna, and Peter Skands. A brief introduction to pythia 8.1. Computer Physics Communications , 178(11):852-867, Jun 2008.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 12438-12448. Curran Associates, Inc., 2020.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning , ICML '08, page 1064-1071, New York, NY, USA, 2008. Association for Computing Machinery.

Mary Touranakou, Nadezda Chernyavskaya, Javier Duarte, Dimitrios Gunopulos, Raghav Kansal, Breno Orzari, Maurizio Pierini, Thiago Tomei, and Jean-Roch Vlimant. Particle-based fast jet simulation at the lhc with variational autoencoders, 2022.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017.

Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning , ICML'11, page 681-688, Madison, WI, USA, 2011. Omnipress.

Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning , pages 2635-2644. PMLR, 2016.

Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14971-14980, 2021.

Sangwoong Yoon, Yung-Kyun Noh, and Frank Chongwoo Park. Autoencoding under normalization constraints. In International Conference on Machine Learning , 2021.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alex Smola. Deep sets. In NIPS , 2017.

Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International Conference on Machine Learning , 2016.

A Experimental Details

Datasets For the standard EBM, we train on 300,000 simulated QCD jets. For the hybrid model EBM-CLF, we train on 300,000 simulated Standard Model jets (100,000 QCD jets, 100,000 boosted jets originating from the W boson, and 100,000 boosted jets originating from the top quark). For OOD detection test sets, we employ the hypothetical Higgs boson (in the decay mode of H → hh → ( b ¯ b )( b ¯ b ) ) with a mass of 174 GeV, which decays into two lighter Higgs bosons of 80 GeV. The intermediate light Higgs boson decays into two b quarks. Each test set consists of 10,000 samples that are p T -refined to the range of [550, 650] GeV. All the jet samples are generated with a pipeline of physics simulators.

Event Generation QCD jets are extracted from QCD di-jet events that are generated with MadGraph [4] for LHC 13 TeV, followed by Pythia8 [61] and Delphes [18] for parton shower and fast detector simulation. All jets are clustered using the antik T algorithm [9] with cone size R = 1 . 0 and a selection cut in the jet transverse momentum p T > 450 GeV. We use the particle flow objects for jet clustering.

Input Preprocessing Jets are preprocessed before being fed into the neural models. Jets are longitudinally boosted and centered at (0 , 0) in the ( η, ϕ ) plane. The centered jets are then rotated so that the jet principal axis ( ∑ i η i E i R i , ∑ i ϕ i E i R i ) (with R i = √ η 2 i + ϕ 2 i ) and E i is the constituent energy) is vertically aligned on the ( η, ϕ ) plane.

Hyper-parameters Hyper-parameters are recorded in Table 4. Hyper-parameters for the transformer are chosen according to a jet classification task. We scan over the following hyper-parameter ranges:

We found that fewer steps {5, 10} make training unstable. Thus the model prefers a relatively larger number of steps and a smaller step size. However, 60 steps MCMC takes much longer time to train and no significant improvement was observed for even longer chains. We balance training efficiency and effective learning by choosing 24 steps.

The MCMC chains are initialized with Gaussian noises, where the constituent features are sampled from the following distributions: log p Ti ∼ N (2 , 1) , η i ∼ N (0 , 0 . 1 2 ) , and ϕ i ∼ N (0 , 0 . 2 2 ) .

Table 4: Model settings.

B Additional Results

B.1 Ablation Study

We explore the most crucial aspects of the model to test their functionality:

With an energy function approximated with a Multi-Layer-Perceptron (MLP) net, we were not able to achieve quality generation.

We also tried out the Hamiltonian Monte Carlo (HMC) [52] for the MCMC procedure. However, we were not able to achieve good performance in these experiments.

Results measured in the Jensen-Shannon Divergence of the high-level observables ( p T and M ) are recorded in Table 5.

0.040

Data

Data

Model Generation

Model Generation

0.035

0.005

0.030

0.004

0.025

0.003

0.020

a

a

0.015

0.002

0.010

0.001

0.005

0.000

0.000

0

200

400

600

800

1000

0

100

200

300

400

pT/GeV

M/GeV

B.2 Classification Performance of EBM-CLF

The classification accuracies for EBM-CLF (QCD/W/Top 3-way classification) are recorded in Table 6. EBMCLF performs on par with the fully supervised classifier ParticleNet [59], while EBM-CLF is trained on a much smaller dataset. The corresponding confusion matrices are displayed in Fig. 6. When we empirically down-weight the term log p ( y | x ) (decrease κ in Eq. 7), the classification performance drops correspondingly.

B.3 Additional Plots

In Fig. 7, we show the generated jet samples displayed in images on the ( η, ϕ ) plane. In Fig. 8, we show the high-level observable distributions of generated jet samples of EBM-CLF. In Fig. 9, we show the background mass distributions under different acceptance rates ϵ after cutting on the energy score of the standard EBM.

0.9

0.8

0.8

QCD

0.81

0.07

0.11

QCD

0.84

0.05

0.10

0.7

0.7

-0.6

0.6

Actual

0.07

0.89

0.04

0.5

Actual

0.06

0.90

0.04

0.5

- 0.4

- 0.4

- 0.3

- 0.3

0.09

0.07

0.84

-0.2

0.07

0.06

0.87

- 0.2

- 0.1

- 0.1

QCD

W

QCD

W

doL

Predicted

Predicted

1.00

1.00

1.00

1.00

Model Generation

1.00

Real Jets

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.00

00'0

0.00

0.00

G-

0.00

0.25

0.25

0.25

0.25

0.25

0.50

0S0

0.50

0.50

0.50

0.75

0.75

0.75

0.75

0.75

1.00

1.0

0.5

0.0

0.5

1.0

1.00

1.0

0.5

0'0

0.5

1.0

1.00

1.0

0.5

0.0

0.5

1.0

1.00

1.0

0.5

0.0

0.5

1.0

1.00

1.0

0.5

0.0

0.5

1.0

7

n

n

k

C Reproducibility Statement

To ensure reproducibility and encourage open-source practice in the HEP community, we release the code implementation in https://github.com/taolicheng/EBM-HEP . The training sets and test sets are accessible at [47, 14]. Due to difficulties in aligning model comparison protocols for different research groups, we thus focus on methods with code publicly available, that serve as credible baselines, for model comparison.

Data

Data

Model Generation

0.014

Model Generation

0.010

0.012

0.008

0.010

0.006

0.008

0.006

0.004

0.004

0.002

0.002

0.000

0.000

0

200

400

009

800

1000

0

100

200

300

400

500

pT/GeV

M/GeV

0

50

100

150

200

250

300

350

MJ

/GeV

10

5

10

4

10

3

10

2

QCD [

=1.0

]

QCD [

=0.5

]

QCD [

=0.2

]

QCD [

=0.1

]

Peer reviews:
comments: SUMMARY:
In this paper, the authors describe an energy-based generative framework for modeling the behavior of elementary particles. This framework can then be used to generate High Energy Physics events, similar to those at LHC, as well as be used for anomaly detection, specifically QCD jets.

STRENGTHS:
The strength of the paper lies in the clarity of writing, technical detail, and interesting application of the proposed methodology.

WEAKNESSES:
The main weakness of the paper is that it is hard to gauge how the improvements of the methodology over the baselines would actually translate in improvements of event detection at LHC.

QUESTIONS:
1. In the paper the line "The results of generative sampling are shown in Appendix C" refers to actually refer to Appendix B.2. Please correct this if it is indeed the case. Also, please describe more the dataset used (for example, does it have class imbalances?) and consider including some confusion matrix or supplementary methods to better compare EBM-CLF to ParticleNet
2. Please include in figure 4 left, the curve for EMB-CLF(E(x)) AUC(Top).
3. Table 2 is a bit hard to read because of the many shades of gray and the inclusion of the background QCD classes. Please reconsider how to show the comparison more clearly.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

comments: SUMMARY:
This paper proposes to use EBMs to model the fundamental interactions between elementary particles. The paper first introduces a set of techniques to enable effective learning of energy functions across elementary particles, including the use of a classifier to model the conditional distributions. The paper then proposes as an architecture to capture permutation invariant interaction between particles. The paper proposes a set of metrics to measure the generation quality of EBMs as a substitution for maximum likelihood estimation and illustrate how EBMs can effectively detect anomalies between particles.

STRENGTHS:
- The paper tackles an interesting and timely problem and explore how generative models (EBMs) can be used in the scientific application of particle dynamics.
- The paper is clearly written and experiments demonstrates the utility of EBMs at both modeling the behaviors of underlying particles and detecting spurious patterns
- The introduction compelling introduces and motivates the problem studied by the paper
- In comparison to many other generative models, EBMs are much less well studied,  and this paper represents an original and interesting application of EBMs to different use cases 
- The method section has a variety of interesting modifications and tricks over prior work (such as the removal of the entropy term) that are interesting to read

WEAKNESSES:
- It might be interesting to see the extent to which the energy function learned by EBM correlates to real physical energy.
- It might also be interesting to structure/augment the energy function with ground truth physical energy functions

QUESTIONS:
See weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This work applies recent EBM techniques to model jet streams from the LHC. The work adopts a transformer architecture for the EBM to allow a permutation invariant representation that captures high order relations between particles. EBM models of LHC particles are learned using techniques derived from recent image EBM works. Samples generated from the model are shown to closely match statistics of the observed data, and OOD detection with the learned EBM is performed and shown to have superior performance compared to VAE methods.

STRENGTHS:
* This paper is the first to explore EBM learning in the area of high-energy physics. It is exciting and interesting to see research into EBMs for new and challenging domains.
* Experimental results provide convincing evidence of the efficacy of the learned model for generation and OOD.

WEAKNESSES:
* The paper lacks theoretical and methodological novelty. The work primarily focuses on applications of EBMs to a new domain.
* It appears that the primary practical application of the paper is OOD detection. In this case, it might be more effective to use an EBM designed specifically for OOD, as in [a]. I am not sure use of a generative EBM is well-justified. Tailoring the methodology more towards OOD would increase the focus of the paper, which tends to wander somewhat.

[a] https://arxiv.org/pdf/2010.03759.pdf

QUESTIONS:
* In the view of the authors, what are the most important novel aspects of their work?

LIMITATIONS:
Limitations were not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the use of energy based models to modeling the distribution of jets in high energy particle physics. Here, the data consists of a vector of events, each of which is described by a momentum transverse to the beam and 2D spatial coordinates. The goal is to learn the true distribution of jets from real data via an energy based model, which can subsequently be used for Out-Of-Distribution detection and classification, in the manner of the paper \"Your classifier is secretly an energy based model and you should treat it like one\".\n\nThe model is trained with gradient ascent, using Langevin sampling to estimate the intractable expectation in the loss function, and making use of the (persistent) Contrastive Divergence of the loss. An additional KL term to minimize the difference between the true model distribution and the Langevin approximation is used.\n\nSTRENGTHS:\nThe paper appears to achieve success in a difficult domain of application. Their validation suggests that the energy based model they train successfully captures the desired data distribution, at least up to well-known features. Furthermore, they show that the energy based model can be altered to a supervised setting, and used as a classifier. \n\t\t\nThe use of the transformer as the neural architecture to model inter-particle interactions also appears to be a successful choice.\n\nWEAKNESSES:\nThe ideas in this paper were often conveyed poorly, and it was quite hard to understand the technical details, either of the energy based model training procedure, or of the scientific problem in question. \n\nIn equation 2, x+ and x- were not clearly explained, making the paper hard to read for someone unfamiliar with the details of contrastive divergence. Using the expectation notation of (in Latex) $E_{x^+ \\sim p_D}$ would have helped clarify that the x+ are drawn from the data distribution and the x- from the (approximation of) the model distribution.\n\nAlgorithm 1 describes the \"stopping gradient\", but more information is needed here.\n\nThe equation in Figure 1 are blurry - it needs to be redone in higher resolution.\n\nMore explanation is needed for the anomaly detection task. I did not understand the set up in adequate detail. You say: \"With an energy-based model, which is trained on the QCD background events or directly on the slightly signal-contaminated data, we expect unseen signals to have higher energies and correspondingly lower likelihoods\". But what is a signal vs a background event here? This needs to be clear for a non-HEP audience.\n\nWhat the labels are in the classification task is unclear.\n\nThe paper notes that \"A more intriguing property of EBMs is that spurious correlations can be better handled\", but it wasn't clear to me either what spurious correlations referred to here, or why EBMs would be better.\n\n\"More interestingly, compositionality can be easily incorporated within the framework of EBMs by simply summing up the energy functions \": while I understand what is meant here, the word \"compositionality\" means many things in different contexts, and is too vague. While other papers on EBMs have used \"compositionality\" in this sense, you should either cite them, or explain directly what you mean by it.\n\nQUESTIONS:\nWhich are the original contributions of the paper on the machine learning side, if any? In particular, is the particular version of Contrastive Divergence used, (in the section KL Divergence-Improved EBM Training) key to your success here? I'd like to see more discussion of this KL term, which is quite interesting. \n\nI would be interested to know how this paper fits into a larger research project. Are there further directions that could be pursued, in terms of applications? Did the authors find the approach satisfactory, or were there limitations that concerned them? \n\nSuggestions:\n\nI was able to understand the structure of the paper fully only after reading the following two papers: https://arxiv.org/pdf/1912.03263.pdf and https://arxiv.org/pdf/1903.08689.pdf. The first describes the three applications of energy models you are interested in (generative modeling, OOD detection, and classification). The second describes the Contrastive divergence approach, and cites detailed notes which explain its derivation; your training algorithm appears to be a modification of their Algorithm 1. I would summarize your paper as \"an application of the approach of these papers to HEP Jet modeling\". While you do cite both of these papers, I would recommend highlighting them more, since they form the backbone of your approach and will help the reader to follow along.\n\nLIMITATIONS:\nAre there ways of incorporating domain specific models of HEP into the current approach? Currently it seems that the physics incorporated into simulators is not leveraged here. It would also be interesting if known expressions or heuristics for the energy could be incorporated into the energy function.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}