Particle Cloud Generation with Message Passing Generative Adversarial Networks
Authors: Raghav Kansal
Javier Duarte
Hao Su
Breno Orzari
Thiago R F P Tomei
Maurizio Pierini
Mary Touranakou
Jean-roch Vlimant
Dimitrios Gunopulos
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models
gans
point clouds
physics

Abstract: In high energy physics (HEP), jets are collections of correlated particles produced ubiquitously in particle collisions such as those at the CERN Large Hadron Collider (LHC). Machine learning (ML)-based generative models, such as generative adversarial networks (GANs), have the potential to significantly accelerate LHC jet simulations. However, despite jets having a natural representation as a set of particles in momentum-space, a.k.a. a particle cloud, there exist no generative models applied to such a dataset. In this work, we introduce a new particle cloud dataset (JetNet), and apply to it existing point cloud GANs. Results are evaluated using (1) 1-Wasserstein distances between high- and low-level feature distributions, (2) a newly developed Fréchet ParticleNet Distance, and (3) the coverage and (4) minimum matching distance metrics. Existing GANs are found to be inadequate for physics applications, hence we develop a new message passing GAN (MPGAN), which outperforms existing point cloud GANs on virtually every metric and shows promise for use in HEP. We propose JetNet as a novel point-cloud-style dataset for the ML community to experiment with, and set MPGAN as a benchmark to improve upon for future generative models. Additionally, to facilitate research and improve accessibility and reproducibility in this area, we release the open-source JetNet Python package with interfaces for particle cloud datasets, implementations for evaluation and loss metrics, and more tools for ML in HEP development.

Particle Cloud Generation with Message Passing Generative Adversarial Networks
Raghav Kansal, Javier Duarte, Hao Su University of California San Diego La Jolla, CA 92093, USA
Breno Orzari, Thiago Tomei Universidade Estadual Paulista
São Paulo/SP CEP 01049-010, Brazil
Maurizio Pierini, Mary Touranakou" European Organization for Nuclear Research (CERN) CH-1211 Geneva 23, Switzerland
Jean-Roch Vlimant
California Institute of Technology
Pasadena, CA 91125, USA
Dimitrios Gunopulos
National and Kapodistrian University of Athens
Athens 15772, Greece
Abstract
In high energy physics (HEP), jets are collections of correlated particles produced ubiquitously in particle collisions such as those at the CERN Large Hadron Col- lider (LHC). Machine learning (MI.)-based generative models, such as generative adversarial networks (GANs), have the potential to significantly accelerate LHC jet simulations. However, despite jets having a natural representation as a set particles in momentum-space a.k.a. particle cloud, there exist generative models applied to such dataset. In this work, we introduce new particle cloud dataset (JetNet), and apply it existing point cloud GANs. Results are evaluated using (1) I-Wasserstein distances between high- and low-level feature distributions, (2) newly developed Fréchet ParticleNet Distance, and (3) the coverage and (4) minimum matching distance metrics. Existing GANs are found to be inadequate for physics applications, hence develop new message passing GAN (MPGAN), which outperforms existing point cloud GANs on virtually every metric and shows promise for in HEP. We propose JetNet as novel point-cloud-style dataset for the MI. community to experiment with, and set MPGAN as a benchmark to improve upon for future generative models. Additionally, facilitate research and improve accessibility and reproducibility in this area, we release the open-source JETNET Python package with interfaces for particle cloud datasets, implementations for evaluation and loss metrics, and more tools for ML in HEP development.
Introduction
Over the past decade, machine learning (ML) has become the de facto way to analyze jets, collimated high-energy sprays particles [1] produced the CERN Large Hadron Collider (LHC). To apply ML to jets, the most natural representation is a particle cloud, variable-sized set of points in momentum space, whose radiation pattern contains rich information about the underlying physics known as quantum chromodynamics (QCD). A fundamental question is whether MI. algorithms can model this underlying physics and successfully reproduce the rich high- and low-level structure in jets.
Also at National and Kapodistrian University of Athens, Athens, Greece.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Answering this question affirmatively has important practical applications. At the LHC, large simulated data samples of collision events' are generated using Monte Carlo (MC) methods in order to translate theoretical predictions into observable distributions, and ultimately perform physics analyses'. These samples, numbering in the billions of events, require computationally expensive modeling of the interaction of particles traversing the detector material. Recently developed generative frameworks in ML such as generative adversarial networks (GANs), if accurate enough, can be used to accelerate this simulation by potentially five orders of magnitude [2].
In this work, we advocate for benchmark jet dataset (JetNet) and propose several physics- and ceapputer-iiiooinnpppaee
metries with which the ML community can improve and evaluate generative models in high energy physics (HEP). To facilitate and encourage research in this area, as well as to make such research more accessible and reproducible, we release interfaces for public particle cloud datasets such as JetNet, implementations for our proposed metries, and various tools for MI. in HEP development in the JETNET library [3]. We next apply existing point cloud GANs on JetNet and find the results to be inadequate for physies applications. Finally, we develop our own message passing GAN (MPGAN), which dramatically improves results on virtually every metric, and propose it as benchmark on JetNet.
2 Jets
High-energy proton-proton collisions the LHC produce elementary particles like quarks and gluons, which cannot be isolated due to the QCD property of color confinement [4]. These particles continuously radiate or "split" into a set of particles, known as a parton shower. Eventually they cool to an energy at which they undergo the process of hadronization, where the fundamental particles combine form more stable hadrons, such as pions and protons. The final set of collimated hadrons produced after such process is referred to as a jet.
The task of simulating a single jet can be algorithmically defined as inputting an initial particle, which produces the jet, and outputting the final set of particles a.k.a. the jet constituents. Typically in HEP the parton shower and hadronization are steps that are simulated sequentially using MC event generators such as PYTHIA [5] or HERWIG IG [6]. Simulating either process exactly is not possible because of the complex underlying physics (QCD), and instead these event generators fit simplified physics-inspired stochastic models, such as the Lund string model for hadronization [7], to existing data using MC methods. The present work can be seen as an extension of this idea, using a simpler, ML-based model, also fitted to data, for generating the jet in one shot, where we are effectively trading the interpretability of the MC methods for the speed of GPU-accelerated ML generators.
Representations. As common for collider physics, we use a Cartesian coordinate system with the 2 axis oriented along the beam axis, the x axis the horizontal plane, and the y axis oriented upward. The & and axes define the transverse plane, while the = axis identifies the longitudinal direction. The azimuthal angle is computed with respect to the axis. The polar angle 0 is used to compute the pseudorapidity 1) = - log(tan(() (9/2)). The transverse momentum (pT) is the projection of the particle momentum the (x, y) plane. As is customary, we transform the particle momenta from Cartesian coordinates (px, Pz) to cewpuddiinn--oo-iiaanat pseudo-angular coordinates (PT, 17, Φ), shown in Fig.
In the context of ML, jets can be represented in multiple ways. One popular representation is as images [8, 91. created by projecting each jet's particle constituents onto a discretized angular n-- plane, and taking the intensity of each "pixel" in this grid to be a monotonically increasing function of the corresponding particle Pr. These tend to be extremely sparse, with typically fewer than 10% of pixels nonempty [10], and the discretization process can furthemore lower the resolution.
Two more spatially efficient representations are as ordered lists or unordered sets [11, 12] of the jet constituents and their features. The difficulty with the former is that there is no particular preferred ordering of the particles-on would have to impose an arbitrary ordering such by transverse momentum [13]. The more natural representation is the unordered set of particles in momentum space, which refer to "particle cloud." This is in analogy to point cloud representations of 3D
2 An event a set observed particles with well-defined momenta, but not generally well-defined positions. "See App. A for further details on the downstream applications of simulations.

---

1,32
0.58
7 0.55
jet 200s
t-
Figure 1: The collider physics coordinate system defining (pT: 1), Φ) (left). The three jet classes in our dataset (right). Gluon (g) and light quark (q) jets have simple topologies, with q jets generally containing fewer particles. Top quark (t) jets have a complex three-pronge structure. Shown also are the relative angular coordinates 177e and ørel measured from the jet axis.
objects in position-space prevalent in computer vision created, for example, by sampling from 3D ShapeNet models [14].
Apart from how the samples are produced, significant differences between jets and ShapeNet-based point clouds are that, firstly, jets have physically meaningful low- and high-level features such as particle momentum, total mass of the jet, the number of sub-jets, and n--particle energy correlations. These physical observables are how we characterize jets, and hence are important to reproduce correctly for physics analysis applications. Secondly, unlike the conditional distributions of points given particular ShapeNet object, which are identical and independent, particle distributions within jets are highly correlated, as the particles each originate from a single source. The independence of their constituents also means that ShapeNet-sample point clouds can be chosen to be of fixed cardinality, whereas this is not possible for jets, which inherently contain varying numbers of particles due to the stochastic nature of particle production.
JetNet. We publish JetNet [15] under the CC-BY 4.0 license, to facilitate and advance ML research in HEP, and to offer new point-cloud-style dataset to experiment with. Derived from Ref. [16]*, it consists of simulated partiele jets with transverse momenta y²r TeV, originating from gluons, light quarks, and top quarks produced in 13 TeV proton-proton collisions in a simplified detector. Technical details of the generation process are given in App. B. We limit the number of constituents to the 30 highest pT particles per jet, allowing for jets with potentially fewer than 30 by zero- padding. For each particle we provide the following four features: the relative angular coordinates nfte| = Paraticce - splct and rrel = @particke Eest (mod 2T), relative transverse momentum pir p partiicl /ple* and binary mask feature classifying the particle as genuine zero-patee eero-padded.
We choose three jet classes, depicted in Fig. to individually target the unique and challenging properties of jets. Gluons provide a useful baseline test, as they typically radiate into a large number of particles before hadronization, largely avoiding the variable-sized cloud issue-ar at least with a 30 particle maximum, and have relatively simple topology. Light quarks share the simple topology, but produce fewer final-state particles, resulting in a larger fraction of zero-padded particles in the dataset. They allow evaluation of model's ability to handle variable-sized clouds. Finally, top quarks decay into three lighter quarks through an intermediate particle, the W boson, which each may produce their sub-jets, leading to complex two- three prooneed topology-depee depending clustering algorithm captures all three or just two of these sub-jets. This results bimodal jet feature distributions (one peak corresponding to fully merged top quark jets and the other to semi-merged, as seen in Fig. 3). Thus, top quark jets test models' ability to learn the rich global structure and clustering history of a particle cloud.
whether the jet
3 Related Work
Generative models in HEP. Past work in this area has exclusively used image-based represen- tations for HEP data. One benefit of this is the ability to employ convolutional neural network (CNN) based generative models, which have been highly successful on computer vision tasks.
"This dataset also released under the CC-BY 4.0 license.
3

---

Refs. [2. 17-20), for example, build upon CNN-based GANs, and Ref. [21] uses an auto rugr-regressive model. to output jet- and detector-data-imen mata-ssages.
In addition to the issues with such representations outlined in Sec. 2, the high sparsity of the images can lead training difficulties in GANs, and the irregular geometry of the data single LHC detector can typically have multiple sections with differing pixel sizes and shapes - poses a challenge for CNN GANs which output uniform matrices. While these can be mitigated to an extent with techniques such as batch normalization [22] and using larger/more regular pixels [18], our approach avoids both issues by generating particle-cloud partcle-cloud-repressss rloudrrepresentatioon of the data, as these are inherently sparse data structures and are completely flexible to the underlying geometry.
GANs for point clouds. There several published generative models in this area, however the majority exploit inductive biases specific to their respective datasets, such as ShapeNet-based [23-26] and molecular [27-29] point clouds, which are not appropriate for jets. A more detailed discussion, including some experimental results, can be found in App. C.
There do exist some more pereroe-purpose GAN models, namely r-GAN [30], GraphCNN-GAN [31], and TreeGAN [32], and we test these on JetNet. r-GAN uses a fully-connected (FC) network, GraphCNN-GAN uses graph convolutions based on dynamic k-nn graphs in intermediate feature spaces, and TreeGAN iteratively up samples the graphs with information passing from ancestor to descendant nodes. In terms of discriminators, past work has used either FC or PointNet [33]-style network. Ref. [34] is the first work to study point cloud discriminator design in detail and finds amongst number of PointNet and graph convolutional models that PointNet-Mix, which uses both max- and average-pooled features, is the most performant.
We apply the three aforementioned generators and FC and PointNet-Mix discriminators as baselines to our dataset, but find jet structure is not adequately reproduced. GraphCNN's local convolutions make learning global structure difficult, and while the TreeGAN and FC generator + PointNet discriminator combinations are improvements, they are not able to learn multi caltiipprticle correlations, particularly for the complex top quark jets, nor deal with the variable-sized light quark jets to the extent necessary for physics applications.
Message Passing Neural Networks. We attempt to overcome limitations of existing GANs by designing a novel generator and discriminator which can learn such correlations and handle variable- sized particle clouds. Both networks build upon the generic message-passineg passing neural network (MPNN) [35] framework with physics-conscionn design choices, and collectively we refer to them as mussage-passing GAN (MPGAN). We find MPGAN outperforms existing models virtually all evaluation metries.
3.1 Evaluating generative models.
Evaluating generative models is a difficult task, however there has been extensive work in this area in both the physics and computer-vision communities.
Physics-inspired metries. An accurate jet simulation algorithm should reproduce both low-level and high-level features (such as those described in See. 2), hence standard method of validating generative models, which we employ, is to compare the distributions of such features between the real and generated samples [2, 17-20.36].
For application in HEP, a generative model needs to produce jets with physical features indistin- guishable from real. Therefore, we propose the validation criteria that differences between real and generated sample features may not exceed those between sets of randomly chosen real samples. To verify this, we use bootstrapping to compare between random samples of only real jets as a baseline.
practically useful set of features to validate against are the so-called "energy-flow polynomials" (EFPs) [37], which are set of multi-particl correlation functions. Importantly, the set of all EFPs forms linear basis for all useful jet-level features Therefore, we claim that if we observe all EFP distributions to be reproduced with high fidelity and to match the above criteria, we can conclude with strong confidence that our model is outputting accurate particle clouds.
$ Moor details on the downstream validation procedure used HEP analyses can be found in App. "I the context of HEP, this means all infrared- and colinear-safe observables.

---

Numder of parriiioss neneont surper
MP()FFC Generator
Croaated Binde
Schertndggg
Fra
Taalo asger ad fris ff
Paeing
Coud
Genersee Perice Cloid
Cononted
MP Discriminator
Figure 2: Top: The MP generator uses message passing to generate a particle cloud. In blue is the initial latent vector and FC layer part of the MP-LFC variant. Bottom: The MP discriminator uses message passing to classify an input particle cloud as real or generated.
Computer-vision -insii-d metries A popular metric for evaluating images which has shown be sensitive to output quality and mode-collapse, though it has its limitations [38], is the Fréchet Inception Distance [39] (FID). FID is defined as the Fréchet distance between Gaussian distributions fitted to the activations of fully-connected layer of the Inception-v3 image classifier in response to real and generated samples. We develop a particle-cloud-ee -lonaaallo--d of this metric, which we call Fréchet ParticleNet Distance (FPND), using the state-of -ortthe-rrtttttt- (SOTA) ParticleNet graph convolutional jet classifier [10] lieu the Inception network. We note that the FPND and comparing distributions as above is conceptually equivalent, except here instead of physically meaningful and easily interpretable features, we are comparing those found to be statistically optimum for distinguishing jets.
Two common metrics for evaluating point cloud generators are coverage (COV) and minimum matching distance (MMD) [30]. Both involve finding the closest point cloud in a sample X each cloud in another sample Y. based on a metric such as the Chamfer distance or the earth mover's distance. Coverage is defined as the fraction of samples in X which were matched to one in Y. measuring thus the diversity of the samples in Y relative to X, and MMD is the average distance between matched samples, measuring the quality of samples. We use both, and due to drawbacks of the Chamfer distance pointed out in Ref. [30], for our distance metric choose only the analogue of the earth mover's distance for particle clouds a.k.a. the energy mover's distance (EMD) [40]. We discuss the effectiveness and complementarity of all four metrics in evaluating clouds in Sec. 5.
4 MPGAN Architecture
We describe now the architecture of our MPGAN model (Fig. 2), noting particle aspects compared its r-GAN and GraphCNN-GAN predecessors.
Message passing. Jets originate from a single source particle decaying and hadronizing, hence they end up with important high-level jet features and a rich global structure, known as the jet substructure [1]. stemming from the input particle. Indeed any high-level feature useful for analyzing jets, such as jet mass or multi-particle correlations, necessarily global [37]. Because of this, while past work in learning on point clouds [10,41 42), including GraphCNN-GAN, has used a locally
5

---

connected graph structure and convolutions for message passing. we choose fully connected graph, equally weighting messages from all particles in the clouds. Rather than subtracting particle features for messages between particles, useful in graph convolutions to capture local differences within neighborhood, the respective features are concatenated to preserve the global structure (the difference between particle features is also only physically meaningful if they are in the 4-vector representation of the Lorentz group). During the update step in the message passing we find it empirically beneficial to incorporate a residual connection to previous particle features.
The operation can be described as follows. For an N N-particle cloud J {pi.... pix after iterations of message passing, with t = 0 corresponding to the original input cloud, each particle pl represented by features hf. One iteration of message passing then defined as
m = =f( (h) @ h,),
(1)
h = ff+ (h. + Em
(2)
JEJ
where m. 1++1 is the message vector sent from particle j to particle h are the updated features of particle and fe f++11 and ft+1 are arbitrary functions which, in our case, are implemented as multilayer perceptrons (MLPs) with 3 FC layers.
particle cloud for the MPGAN generator: (1) directly initializing the cloud with N partieles with L randomly sampled features, which refer to as the MP generator, and (2) inputting a single Z-dimensional latent noise vector and transforming it via an FC layer into N L-dimensional matrix, which refer to the MP-Latent-FC (MP-LFC) generator. The MP-LFC uses a latent space which can intuitively be understood as representing the initial source particle's s features along with parameters to capture the stochasticity of the jet production process. Due to the complex nature of this process, however, we posit that this global, flattened latent space cannot capture the full phase space of individual particle features. Hence, introduce the MP generator, which samples noise directly per particle, and find that it outperforms MP-LFC (Table 2).
Generator. We test two initializations
Discriminator: We find the MP generator, in conjunction with a PointNet discriminator, to a significant improvement on every metric compared to FC and GraphCNN generators. However, the jet-level features are not yet reproduced to high enough accuracy (Sec. 5). While PointNet is able to capture global structural information, it can miss the complex interparticle correlations in real particle clouds. We find we can overcome this limitation by incorporating message passing in the discriminator well as in the generator. Concretely, or generated cloud and applies MP layers to produce intermediate features for each particle, which are then aggregated via fcature-wise average- paoragg-pooiing operation and passed through FC layer to output the final scalar feature. We choose 2 MP layers for both networks.
MP discriminator receives the real
Variable-sized clouds. In order to handle clouds with varying numbers of particles, as typical of jets, we introduce an additional binary "masking" particle feature classifying the particle as genuine or pardedd Particles in the cero-padded class are ignored entirely in the message passing and pooling operations. The MP generator adds mask features to the initial particle cloud, using an additional input of the size of the jet N, sampled from the real distribution, before the message passing layers based on sorting in particle feature space. Ablation studies with alternative (as well as no) masking strategies are discussed in App. E.
5 Experiments
Evaluation. We use four techniques discussed in Sec. 3.1 for evaluating and comparing models. Distributions of physical particle and jet features are compared visually and quantitatively using the Wasserstein-1 (W₂) distance between them. For ease of evaluation, we report (1) the average scores of the three particle features (WP) 17te 5Tel, and prl (2) the jet mass (WM). and (3) the average of subset of the EFPs? (WEFP), ), which together provide a holistic picture of the low- and high-level aspects of a jet. The W1 distances are calculated for each feature between random samples
We choose 5 EFPs corresponding the set of loopless multigraphs with 4 vertices and 4 edges.

---

Table 1: W1 distances between real jet mass (WM). averaged particle features (WT), ) and averaged jet EFPs (WTFP) distributions calculated as baseline, for three classes of jets.
Jet class
WM (x10-3)
WP (x10-9)
WPFP (x10-9)
Gluon
0.7
0.44 0.09
0.62 = 0.07
Light quark
0.5 0.1
0.5 0.1
0.46 + 0.04
Top quark
0.51 + 0.07
0.55 + 0.07
1.1 0.1
of 10,000 real and generated jets, and averaged over batches. Baseline W1 distances are calculated between of randomly sampled real jets with 10,000 samples each, and are listed for each feature Table 1. The real samples are split 70/30 for training/evaluation. We train ParticleNet for classification
dataset to develop the FPND) metric. FPND is calculated between 50,000 random real and generated samples, based on the activations of the first FC layer in our trained model". Coverage and MMD are calculated between 100 real and 100 generated samples, and averaged over 10 such batches. Implementations for all metrics are provided in the JETNET package [3].
32
21
2.0
630
1.2
Patcee"
Puticke p?
-0C
100
Parish 1
Pariile of
3.00
6.00
6.2
Comparison of real and generated distributions for subset of jet and partiele features. We use the best performing model for each of the FC, GraphCNN, TreeGAN, and MP generators, as per Table 2. Top: gluon jet features, Middle: light quark jets, Bottom: top quark jets.
Figure 3:
Results. On each of JetNet's three classes, we test r-GAN's FC, GraphCNN, and TreeGAN gener- ators with rGAN's FC and the PointNet-Mix discriminators, and compare them MPGAN's MP generator and discriminator models, including both MP and MP-LFC generator variations. Training and implementation details for each can be found in App. D, and all code in Ref. [43].
We choose model parameters which, during training, yield the lowest W. score. This is because (1) Wi scores between physical features more relevant for physics applications than the other three metrics, and (2) qualitatively we find it be a better discriminator of model quality than particle features EFP scores. Table 2 lists the scores for each model and class, and Fig. shows plots of selected feature distributions of real and generated jets, for the best performing FC, GraphCNN, TreeGAN,
ParticleNet training details given in App. D.3. The trained model is provided in the JETNET library [3].
7

---

Table 2: Six evaluation different generator and discriminator combinations. Lower is better for all metrics except COV.
W (x10-
WP
WT
Jet class
Generator
Discriminator
FPND
COV f
MMD
(x10-
(x10
FC
FC
18.3 - 0.2
9.4
8.5 0.3
176
0.24
0.045
GraphCNN
FC
2.6 0.2
9.3
12
61
0.39
0.046
TreeGAN
41.9 0.3
0.3
14.2
0.19
0.130
FC
PointNet
1.4
1.3
1.5 0.9
5.0
0.49
0.039
GraphCNN
PointNet
0.2
16+6
200 1000
0.46
0.040
TreeGAN
PointNet
1.1
4.4
4
84
0.37
0.042
Gluon
MP
MP
0.7
0.3
0.7
0.12
0.56
0.037
MP-LFC
MP
0.69
1.8 1.2
0.9 = 0.6
0.20
0.54
0.037
MP
0.3
21.1 0.2
9
368
0.11
0.085
GraphCNN MP
2.1
9.2
1318
61
0.38
0.048
TreeGAN
MP
0.2
1247
18 18
69
0.34
0.048
MP
1.2
3.5
1.6
0.44
0.040
MP
PointNet
1.4
1.4
4
18
0.53
0.036
FC
FC
0.2
0.9
3.9 + 0.6
395
0.18
0.053
GraphCNN
3.2
1..1
0.25
TreeGAN
FC
0.3
2.4
9.3
176
0.06
0.055
FC
PointNet
3.1
4.4
2.6
17
0.37
0.028
GraphCNN
PointNet
4+1
0.5
50k #100k
316
0.37
0.031
Light
TreeGAN
PointNet
1.11
5.5
4.1 + 0.3
11
0.47
0.031
quark
MP
MP
0.2
0.5
0.7 + 0.4
0.35
0.50
0.026
MP-LFC
MP
0.2
2.4
0.9
0.08
0.52
0.024
FC
MP
0.2
0.2
4.0
0.11
0.070
GraphCNN
MP
3.5
15.0 0.3
10 10
99
0.26
0.038
TreeGAN
MP
0.2
33
1042
0.22
MP
FC
1.3
4.5
2.2 0.6
41
0.37
0.030
MP
PointNet
0.3
2.62
6
850
0.18
0.034
FC
0.3
14.5
23 3
160
0.28
0.103
GraphCNN
FC
0.3
8.0
16k
0.48
0.081
TreeGAN
FC
17.0
1.66
33 2
77
0.39
0.083
FC
PointNet
2.1
1.6
0.5
0.56
0.075
GraphCNN
PointNet
11.3 0.9
10
37
30k
0.39
0.085
Top
TreeGAN
PointNet
5.19 0.08
9.1
16+22
0.53
0.079
quark
MP
MP
0.2
0.3
211
0.37
0.57
MP-LFC
MP
0.9
2.2+0.7
2+1
0.93
0.56
0.073
FC
MP
6.9
39.1 +.3
15
81
0.26
0.120
GraphCNN
MP
6.7
0.2
40 10
0.49
0.081
TreeGAN
MP
1.44
45
50
66
0.29
0.101
MP
FC
12.9 0.3
2.43
46
58
0.27
0.103
MP
PointNet
0.76
1.6
411
0.59
0.072
and MP generators. We also provide in App. discretized images in the www.ulla-ccoordiats-pppip a.k.a "jet images", however, we note that it is in general not easy visually evaluate the quality of individual particle clouds, hence we focus on metrics and visualizations aggregated over batches of clouds. Overall we find that MPGAN is a significant improvement over the best FC, GraphCNN and TreeGAN models, particularly for top and light quark jets. This is evident both visually and quantitatively in every metric, especially jet W, and FPND, with the exception of WP where only the FC generator and PointNet discriminator (FC PointNet) combination is more performant.

---

We additionally perform latency measurement and find, using an NVIDIA A100 GPU, that MPGAN generation requires 35.7 jus jerjet. comparison, the traditional generation process for JetNet is measured on an 8-CPU machine as requiring 46ms per jet, meaning MPGAN provides a three-orders- of-magnitude speed-up. Furthermore, as noted in App. B, the generation of JetNet is significantly simpler than full simulation and reconstruction used at the LHC, which has been measured to require 12.3s [44] and 4s [45] respectively per top quark jet. Hence in practical applications we anticipate MPGAN's improvement to potentially rise to Eie-orders-of-mmanniuuee
Real baseline comparison. We find that MPGAN's jet-level Wi scores all fall within error of the baselines in Table I, while those of alternative generators several standard deviations away. This is particularly an issue with complex top quark particle clouds, where we can see in Fig. 3 none of the existing generators are able to learn the bimodal jet feature distributions, and smaller light quark clouds, where we see distortion of jet features due to difficulty reproducing the zero-padded particle features. No model is able achieve particle-level scores close to the baseline, and only those of the FC + PointNet combination and MPGAN are of the same order of magnitude. We conclude that MPGAN reproduces the physical observable distributions to the highest degree of accuracy, but note, however, that it requires further improvement in particle feature reconstruction before it is ready for practical application in HEP.
Architecture discussion. To disentangle the effectiveness of the MP generator and discriminator, we train each individually with alternative counterparts (Table 2). With the same PointNet discrimina- tor, the GraphCNN and TreeGAN generators perform worse than the simple FC generator for every metric on all three datasets. The physics- motivated MP generator on the other hand outperforms all on the gluon and top quark datasets, and significantly on the jet-level W1 scores and the FPND. We note, however, that the MP generator is not a significant improvement over the other generators with an FC discriminator. Holding the generator fixed, the PointNet discriminator performs significantly better over the FC for all metrics. With the FC, GraphCNN, and TreeGAN generators, PointNet is also improvement over the MP discriminator. With MP generator, the MP discrimimator is more performant on jet-level W, and FPND scores but, on the top quark dataset, degrades WP relative to PointNet.
We learn from these three things: (1) a generator or discriminator architecture is only as effective as its counterpart- even though the MPGAN combination is the best overall, when paired with network which is not able to learn complex substructure, or which breaks the permutation symmetry, neither the generator discriminator is performant, (2) for high-fidelity jet feature reconstruction, both networks must be able to Iearn complex multi-particle correlations however, this can come at the cost of low-level feature accuracy, and (3) MPGAN's masking strategy is highly effective both MP networks are improvements all around on light quark jets.
Particle cloud evaluation metries. We now discuss the merits of each evaluation metrics and provide suggestions for their use in future work. Fig. 4 shows correlation plots between chosen pairs of our evaluation metries. As expected, we find WI-M and WI-EFP to be highly correlated, as they both measure learning of global jet features. For rigorous validation we suggest measuring both but for time-sensitive use-cases, such as quick evaluations during model training, W1-M should be sufficient. W1-M, FPND, and W1-P are all measuring different aspects of the generation and are relatively uncorrelated. We expect FPND overall be the best and most discriminatory metric for evaluation, as it compares features found by SOTA classifier to be statistically optimum for characterizing jets, while the WI scores are valuable for their interpretabilityy Out of these, WI-M/W1 I-EFP are the most important from a mhyssis-sttadpoing features of the output jets, rather than the individual particle features.
we generally characterize collisions by the high-level
MMD and coverage are both valuable for specifically evaluating the quality and diversity of samples respectively, however we see from Fig. 4 that they saturate after a certain point, after which FPND and W1 scores are necessary for stronger discrimination. We also note that in Table 2, models with low W1 scores relative to the baseline have the best coverage and MMD scores as well. This indicates that the W1 metrics are sensitive to both mode collapse (measured by coverage), which is expected as in terms of feature distributions mode collapse manifests as differing supports, to which the W, distance is sensitive, as well as to individual sample quality (measured by MMD), which supports our claim that recovering jet feature distributions implies accurate learning of individual cloud structure. Together this suggests that low W1 scores are able validate sample quality and
9

---

W1. P va FPND Corelaation
W1 va WI-P Correlation
W1 va FPND Correlation
FPNO
Kimbee serchee
1.27
Nurtee of so0ee
309
Numboo
648530
Wi
W1
WT M
Corre
W1-M Comelation
W1-57
Mumbor of bardes
Nambor bochss
Coverge
8,0
3.4
0.20
0.022
0.054
0.254
3.20
0.03
0.03
0.33
3.50
0.008
0.116
5.04
0.54
10:0
0.03
Figure Correlation plots between pairs of evaluation metrics, evaluated on 400 separate hatches of 50,000 MPGAN generated top quark jets.
against mode collapse, and justifies criteria that practical ML simulation alternative have Wi scores close to the baselines Table 2. conclusion, for thorough validation of generated particle clouds, recommend considering all three W-1 scores in conjunction with FPND, while MMD and coverage, being focused tests of these aspects of generation, may be useful for understanding failure modes during model development.
Summary
In this work, we publish JetNet: novel particle cloud dataset to advance machine learning (ML) research in high energy physics (HEP), and provide novel point-cloud-style dataset containing rich underlying physics for the ML community to experiment with. We apply existing state-of-the-ar point cloud generative models to JetNet, and propose several physics- and computer-vision centtter-viiio-nnsppiiii innsireed metrics to rigorously evaluate generated clouds. We find that existing models are not performant on number of metrics, and fail to reproduce high-level jet features arguublly the most significant aspect generative adversarial network (MPGAN) model, designed to
for HEP. Our new message-passing capture complex global structure and handle variable-sized clouds significantly improves performance in this area, as well as other metrics. We propose MPGAN as new baseline model on JetNet and invite others improve upon
Impact With our JetNet dataset and library, we hope lower the barrier to entry, improve repro- ducibility, and encourage development in HEP and ML. Particularly so in the area of simulation, where accurate and fast ML particle cloud generator will have significant impact in (1) lowering the computational and energy cost of HEP research, as well as (2) increasing precision and sensitivity to new physics at the Large Hadron Collider and future colliders by providing more high-quality simulated data samples. One negative consequence of this, however, may be loss of interpretability. and hence trustability, uncertainties- though the metrics we propose should mitigate against this. More broadly, further advancements the field of MI. point cloud generation may result fake visual data generation for proliferation of misinformation and impersooatiionfidentiyy theft.
the particle production generative model, which may ultimately increase
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 772369). R. K. was partially supported by an IRIS-HEP fellowship through the U.S. National Science Foundation (NSF) under Cooperative Agreement OAC- 183660,,,,, and by the LHC Physics Center at Fermi National Accelerator Laboratory, managed and operated by Fermi Research Alliance, LLC under Contract No. DE-ACO2- 07CH1 1359 with the U.S. Department of Energy (DOE). J. D. is supported by the DOE, Office of Science, Office High Energy Physics Early Career Research program under Award No. DE- SC0021 187 and by the DOF, Office of Advanced Scientific Computing Research under Award No. DE-SC0021396 (FAIR4HEP). B. and T. are supported by grant 2018/25225-9 São Paulo Research Foundation (FAPESP). B. O was also partially supported by grants #201 00018//019/91 and #2019/16401-0, São Paulo Research Foundation (FAPESP). J-R. V. is partially supported by the ERC under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 772369) and by the DOE, Office of Science, Office of High Energy Physics under Award No. DE-SC001 1925, DE-SC0019227, and DEEAAC0-07CH11335 by the EU ICT-48 2020 project TAILOR (No. 952215), This work was performed using the Pacific Research Platform Nautilus HyperCluster supported by NSF awards CNS-1730155 ACI- 1540112, ACI-154133 14444, DAC-1826966 the University of California Office of the President, and the University of California San Diego's California Institute for Telecommunications and Information Technology/Qualcomm Institute. Thanks to CENIC for the 100 Gpbs networks. Funding for cloud credits was supported by NSF Award # #1904444 Internet2 supported E-CAS Exploring Clouds to Accelerate Science.
D. G. is partially supported
References
[1] A. J. Larkoski, I. Moult, and B. Nachman, "Jet Substructure the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning.. Phys. Rept. 841 (2020) 1, doi:10. 1016/j j pwyspep.2200 2019. 11. 001, arXiv: 1709. 04464.
[2] L. de Oliveira, M. Paganini, and B. Nachman, "Learning particle physics by example: Location-aware generative adversarial networks for physics synthesis", Comput. Soft: Big Sci.
1 (2017) 4, : 100.11107/41711-11700000000
arXiv: 05927.
[3] R. Kansal, "JETNET Library" doi :tt:::1.:::::..0011.11.111 500.0.5211111/2 www.///000........ http://zmooood.57799333
https://githe hht::/b/g/thtu.../m http:ttt....oooeeeeeeeeeteeee wetpa/jeenet/Jeee
[4] R. K. Ellis, W. J. Stirling, and B. R. Webber, "QCD and collider physics". Cambridge monographs on particle physics, nuclear physics, and cosmology. Cambridge University Press, Cambridge, 2003. Photography by S. Vascotto. doi: 100.1017/CB098B00 100011116888.....
[5] T. Sjöstrand al., "An introduction PYTHIA 8.2", Comput. Phys. Commun. 191 (2015) 159, d01: 10t./0016/// 2015 01. 024, arXiv: 1410 .012..
[6] Bellm et al., "HERWIG 7.0/ ++3.00 release note", Eur. Phys. 76 (2016), no. 4, arXiv:1512 01178. [7] B. Andersson, G. Gustafson, G. Ingelman, and T. Sjostrand, "Parton Fragmentation and String :0.6 1573 (57)))))000 90080-7.
196, doi: :0: htt.:.11.///ppp//100000011----/0
Dynamies", Phys. Rept. 97 (1983) doi: 1001100011000111
[8] J. Cogan, M. Kagan, E. Strauss, and A. Schwarztman, Images: Computer Vision Inspired Techniques for Jet Tagging", JHEP 02 (2015) 118, doi 10. 1 (2015) 118, arXiv: 5675.
[9] I.. de Oliveira et al., -we---leeees-s-eee-_-r - deep learning edition", JHEP 07 (2016) 069,
doi: 10::10.:0 /HTT/ (2016) 1r1100051..05190
[10] H. Qu and L. Gouskos, "ParticleNet: Jet Tagging via Particle Clouds". Phys. Rev. D 101 1056000000......000...........,0 (2020), no. 5, 056019, doi : 10. ..0..1. 1w1.//1yseeee.....m 101 056019, arXiv: 1952.
[11] J. Shlomi, P. Battaglia, and J.-R. Vlimant, "Graph neural networks in particle physics", Mach. Learn.: Sei. Technol. 2 (2021) 021001, doi : 10.880.....88222222111 2153//bbb/a/......
arXiv: 2007. 13681.
[12] Duarte and J.-R. Vlimant, "Graph neural networks for particle tracking and reconstruction".
in Artificial Intelligence for High Energy Physics, P. Calafiura, D. Rousscau, and K. Terao, eds.
11

---

World Scientific Publishing, 12, 2020. arXiv: :011iv:20122. 2012. 01249. Submitted to Int. J. Mod. Phys. A.
doi: :0. 1142/12200.
[13] CMS Collaboration, "Identification of heavy, energetic, hadronically decaying particles using machine-leariing techniques", JINST 15 (2020) P06005, arXiv:2004. 082662.
:oi: 10::000//88/// 01000////7440022215500/////60000
[14] A. X. Chang et al., "ShapeNet: An information-rich 3D model repository", Technical Report, 2015. arXiv:1512.03012
[15] R. Kansal et al., "JetNet", 05, 2021. doi: : 10. wtt.5281//eeooo 4834876..
[16] M. Pierini, J. M. Duarte, N. Tran, and M. Freytsis, "hls4ml LHC jet dataset (30 particles)", 01, 2020. doi: 0tl:::1..8812228 5281/zenodo 3601436.
[17] M. Pagannii, L.. Oliveira, and B. Nachman, "CaloGAN Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks", Phys. Rev. 97 (2018), no. 1, 014021, doi:10. 1103//hyseeee..... 97. 014021,
arXiv:
[18] ATLAS Collaboration, "Fast simulation of the ATLAS calorimeter system with generative adversarial networks", ATLAS Software Public Note ATT--SOFT-PUB--0200000 2020.
[19] M. Erdmann, J. Glombitza, and T. Quast, "Precise simulation electromagnetic calorimeter showers using Wasserstein generative adversarial network", Comput. Softw. Big Sci. (2019)
4, doi: 111.111100000111111111-----00000 7,
arXiv: 1807 01954.
[20] Carminati et al.. "Generative adversarial networks for fast simulation", J. Phys. Conf. Sen. 1088/ 1742 /1t.:///0077772--500/5/111/1/110
1525 (2020) 012064,
[21] Y. Lu, J. Collado, D. Whiteson, and P. Baldi, "Sparse autoregressive models for scalable generation of sparse images in particle physics". Phys. Rev. 103 (2021), no. 3, 036012, :01: 1 1103 ww3///ys/eeD.......// 103 036012, arXiv 2009 14017.
[22] S. loffe and C. Szegedy, "Batch normalization: Accelcrating deep network training by reducing internal covariate shift", Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei, eds., volume 37, p. 448. PMLR, 2015. arXiv: 1502. 03167.
[23] C.-L. Liet al., "Point cloud GAN". in 2019 ICLR Workshop Deep Generative Models for Highly Structured Data. 2018. arXiv: : 1810. 12111.05577 05795.
[24] G. Yang et al., "PointFlow: 3D point cloud generation with continuous normalizing flows", in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) p. 4540. 2019. arXiv: 1906. 12320. doi 10. 1119/ICOCC .219.. .0064..
[25] R. Klokov, E. Boyer, and J. Verbeek, "Discrete point flow networks for efficient point cloud generation", in 2020 European Conference on Computer Vision (ECCV), A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds., p. 694. Springer International Publishing, Cham, 2020. 2017
[26] R. Cai al., "Learning gradient fields for shape generation". in 2020 European Conference on Computer Vision (ECCV), A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds., p. 364. Cham, 2020. arXiv: 2008. 06520.
[27] Köhler, L. Klein, and Noe. "Equivariant flows: Exact likelihood generative learning for
symmetric densities", in Proceedings of the 37th International Conference on Machine Learning, H. D. III and Singh, eds., volume 119 of Proceedings of Machine Learning Research, p. 5361. PMLR, 2020. arXiv: 20066 02425.
[28] G. N. C. Simm, R. Pinsler, G. Csányi, and J. M. Hernández-Lobato "Symmetry-awan actor-critic for 3d molecular design", in International Conference on Learning Representations. 2021.
[29] N. Gebauer, M. Gastegger, and K. Schütt, (yyymmerr--daapleee generation of 3D point sets for the targeted discovery of molecules". in Advances Neural Information Processing Systems 32, H. Wallach et al., eds., p. 7566. Curran Associates, Inc., 2019. arXiv: 00957.
[30] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, "Learning representations and generative models for 3D point clouds", in Proceedings of the 35th International Conference on Machine Learning. J. Dy and A. Krause, eds., volume 80, p. 40. PMLR, 2018.
12

---

[31] D. Valsesia, G. Fracastoro, and E. Magli, "Learning localized generative models for 3D point clouds via graph convolution", in International Conference on Learning Representations (ICLR) 2019. 2019.
[32] D. W. Shu, S. W. Park, and J. Kwon, "3D point cloud generative udversarial network based on tree structured graph convolutions", in 2019 IEEE/CVP International Conference on Computer Vision (ICCV), p. 3858. 2019. arXiv: 06292. doi: 10. 1109/ 1109/ICVV ICCV 2019.00399
[33] C. Qi, H. Su, K. Mo, and L. J. Guibas, "PointNet: Deep learning on point sets for 3D classification and segmentation". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July, 2017.
[34] H. Wang et al., "Rethinking sampling in 3D point cloud generative adversarial networks", 2020. arXiv:2006 07029.
[35] J. Gilmeret al., "Neural message passing for quantum chemistry", in Proceedings of the 34th International Conference on Machine Learning, D. Precup and Y. W. Teh, eds., volume 70,
p. 1263. PMLR, 2017. arXiv: 01212..
[36] CMS Collaboration, "Recent Developments in CMS Fast Simulation", PoS ICHEP2016 (2016) 181, doi 22/22/22221. 282. 0181, arXiv: 1701. 03850.
[37] 'T. Komiske, E. M. Metodiev, and J. Thaler, "Energy flow polynomials: A complete linear basis for jet substructure", JHEP 04 (2018) 013, doi : 10.
10..1......000... 1007 (2018)
arXiv: 17111.11000 07124.
[38] A. Borji, "Pros and cons of GAN evaluation measures: New developments", 2021.
arXiv: 09396.
[39] M. Heuselet al., "GANs trained by two time-scale update rule converge local nash equilibrium", in Advances in Neural Information Processing Systems, Guyon et al., eds., volume 30, p. 6626. Curran Associates, Inc., 2017. arXiv:1706. 08500.
[40] T. Komiske, E. M. Metodiev, and J. Thaler, "Metric Space of Collider Events", Phyy. Rev. Leu. 123 (2019), no. 4, 041801, doi 10. 1w13//1yys0e.eeeeeeee 123..0...00 041801,
arXiv: 1902..990..... 02346.
[41] Y. Wang et al., "Dynamic graph CNN for learning on point clouds", ACM Trans. Graph. 38 (2019) doi :00 1145/3326662, arXiv: 1801. 07829.
[42] Monti et al., "Geometric deep learning on graphs and manifolds using mixture model CNNs".
in 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5425. 2017.
arXiv: 1611. 08402.
doi: 10. 1109//CPR..20 2017
[43] R. Kansal, "MPGAN Code". :0.1::00.00 wtt.:1055222/eeoo..5553300
https:// thtp:///gtttb.... com httpecwwww...oooaanooooeeee
[44] Pedro, Kevin, "Current and future performance of the cms simulation"' EPJ Web Conf. 214 (2019) 02036, doi: 10. 1001/epj /0119/2200000/20220000
[45] C. Chen et al., "Data augmentation the LHC through analysis-specific fast simulation with deep learning", 2020. arXiv: 20100. 01835.
[46] K. Cranmer, J. Brehmer, and G. Louppe, "The frontier of simulation-basee based inference", PNAS 117 (2020) 30055, doi 10. 1.73/ppas.... 1912769117, arXiv: 11::::1111.00000
[47] G. Cowan, "Statistics", Ch. 40 in Particle Data Group, P. A. Zyla et al., "Review of Particle Physics", PTEP 2020 (2020) 083C01, doi: :0.:10.100 1093/ptep/ http://ttppppaaaa...// ptaa104.
[48] CMS Collaboration, "Precise determination of the mass the Higgs boson and tests compatibility of its couplings with the standard model predictions using proton collisions at and TeV", Eur. Phys. 75 (2015) 212, doi :00. 1140/ http///ppp////100000--33333
arXiv: 1411.86666
[49] J. Barlow and C. Beeston, "Fitting using finite Monte Carlo samples", Comput. Phys. Commun. 77 (1993) 219, :00:10.000 :0.0 100.10060010044600 (93) 90005-W.
[50] Brehmer and K. Cranmer, "Simulation-based inference methods for particle physics", Artificial Intelligence for High Energy Physics, P. Calafiura, D. Rousseau, and K. Terao, eds. World Scientific Publishing, 10, 2020. arXiv 2010.06439. Submitted to Inf. J. Mod. Phys.
doi:10 1142/12200
13

---

[51] J. Alwall et al., "The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations", JHEP 07 (2014) 079, doi: 1007 JHEPO7 (2014) 079, arXiv: : 1405. 0301.
[52] R. D. Ball et al., "Parton distributions with LHC data". Nucl. Phys. B 867 (2013) 244-289, :01: 1016/j www.mmullly..b..m 2012. 0033, arXiv: :2277 1303.
[53] P. Skands, S. Carrazza, and J. Rojo, "Tuning PYTHIA 8.1: The Monash 2013 Tune", Eur. Phys.
J. 74 (2014), no. 8, 3024, doi: 10. 1001::00.00000/00011002222200000 -0--0-
arXiv: 1404. 5630.
[54] M. Cacciari, G. P. Salam, and G. Soyez, "The anti-kr jet clustering algorithm", JHEP 04 (2008) 063, doi 10. 1008/ /2001//88880000/0///////63. arXiv: 0802. 1189.
[55] M. Cacciari, G. P. Salam, and G. Soyez, "Fastlet user manual", Eur: Phys. J. C 72 (2012)
1896, doi 10. 1
[56] M. Cacciari and G. Salam, "Dispelling the n myth for the KT jet-finder". Phys. Lett. B 641 (2006) 57, doi 10.1:10::1 101110.. 1016/j physletb. 2006. 08. 037, arXiv:hep 1rXiv:iep-phh0512211 ph/0p22200
[57] X. Mao et al., "Multi-class generative adversarial networks with the I.2 loss function", 2016. arXiv: 04076.
[58] N. Srivastavact al., "Dropout: simple way to prevent neural networks from overfitting". J. Mach. Learn. Res. 15 (2014) 1929.
[59] I. Gulrajaniet al., "Improved training of Wasserstein GANs", in Advances in Neural Information Processing Systems, I. Guyon et al., eds., volume 30, p. 5767. Curran Associates, Inc., 2017. arXiv: 1704. 00028.
[60] Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks", in 6th International Conference on Learning Representations. 2015 8.
arXiv: 1802. 05957.
[61] Schäfer, H. Zheng, and Anandkumar, "Implicit competitive regularization GANs", 2019. arXiv: 1910 05852.
[62] T. Karraset al., "Training generative adversarial networks with limited data", in Advances in Neural Information Processing Systems, H. Larochelle al., eds., volume 33, p. 12104. Curran Associates, Inc., 2020. arXiv: 2.066066766
[63] N.-T. Tran et al., "On data augmentation for GAN training". IEEE Trans. Image Process. 30 (2021) 1882, doi 10. 1109/tip 2021 3049346, arXiv:2006 05338.
[64] Z. Zhao et al., "Image augmentations for GAN training", 2020, arXiv: 02595.
[65] M. Arjovsky, S. Chintala, and L.. Bottou, "Wasserstein GAN", 2017. arXiv: 1701.v:110..07111
14


Reviews:
[{"review_id": "rGVS0tomr3z", "reviewer": "Reviewer_He4i", "summary": "The paper introduces the new particle cloud dataset JetNet and corresponding evaluation metrics for ML in high energy physics. Several existing point-cloud generative models are evaluated and a new baseline model for the dataset is introduced.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The introduction of JetNet is well-motivated and tackles a challenging problem in high energy physics.  The proposal of a message passing GAN (MPGAN) designed for this task shows significant improvement upon existing GAN approaches for the generation of point sets. As the authors note, MPGAN is still not sufficient for real-world applications such that JetNet presents a challenging task that might need novel approaches to achieve sufficient precision with a generative model.\n\nThe paper is clearly written and structured in a way that nicely introduces the problem setting. The discussion of design choices and merits/limitations of the metrics as well as the evaluation of all the models on three different tasks (gluon, light quarks, and top quarks) is thorough.\n\nMinor issues:\n- The comparison focuses very much on GANs. There is also some related work on generating 3d molecules based on autoregressive models and normalizing flows, which deals with similar issues, e.g. variable number of atoms and rotational symmetries (Gebauer et al., NeurIPS 2019, Köhler et al., ICML 2020, Simm et al., ICLR 2021). A discussion on whether these kinds of models might be applicable would be interesting.\n- The fonts in the figures is quite small and hard to read (in particular the labels in Fig 2 and labels in Fig. 4.\n- It would be nice for the order of gluon, light quarks and top quarks to be consistent across plots and tables.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "b43qnSQt78M", "reviewer": "Reviewer_NRE3", "summary": "This paper introduces a new particle cloud dataset (JetNet) for high-energy physics. The authors set up a benchmark for particle cloud generation with a few physics-inspired and vision-related metrics. As existing point cloud GANs are not suitable for physics applications, they have also developed a new message-passing GAN (MPGAN) and have achieved good empirical performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The authors have provided sufficient physics background to help non-physics researchers better understand the problem setup. The topic studied in this paper (using machine learning to advance and accelerate physics research) is very interesting and has a large scientific impact. The authors have promised to release their dataset, which might benefit the researchers in the community a lot.\n\nApart from these mentioned in the related work (L117-L132), there have been a few other papers related to point cloud generation in the literature [Li et al., ICLR 2019 Workshop; Shu et al., ICCV 2019]. The authors have claimed that existing GANs are inadequate for physics applications. However, without sufficient comparisons with these existing methods, this argument is not very well justified. Besides, the authors have also mentioned in the introduction (L25-L31) that ML-based generative models might accelerate the simulation by five orders of magnitude. However, they have not provided any latency measurement on the hardware to support this claim.\n\nThis paper provides a bunch of different metrics for the particle cloud generation. It would be great if the authors could analyze the correlation between different metrics and provide some insights on which metric is the best, which might facilitate the comparisons between different methods in this direction. Besides, it is necessary to provide some qualitative examples to illustrate how the generated samples look like and some discussions on what the target downstream task is. It would be interesting to see how the generated point cloud performs in real physics applications.\n\n**Post-Rebuttal Comments:**\nThe authors' rebuttal has addressed most of my concerns. I especially appreciate the additional results of TreeGAN and PC-GAN. Therefore, I will raise my score by 1 and vote for accepting this paper.\n\nReferences:\n1. Li et al., \"Point Cloud GAN\", ICLR 2019 Workshop.\n2. Shu et al., \"3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions\", ICCV 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not have noticeable negative societal impacts. Besides, the authors have briefly discussed their limitations in Section 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "D3xxmSknGZI", "reviewer": "Reviewer_Q2fq", "summary": "This paper proposes to explore an interesting application of existing point cloud generative models to the problem of generation of jets: high energy sprays of particles from elementary particle colliders. Jets are analogous in a way to point clouds with a difference in considered per-object features: per-particle momenta are considered for jets contrary to per-point 3D coordinates for point clouds. With sufficient level of quality such generative models could potentially be used instead of costly Monte Carlo simulations.\n\nThe authors explore existing point cloud gan-based generative models and propose an adaptation of existing message-passing networks to be used in generators and discriminators of resulting particle cloud gans. Several metrics are proposed for evaluation, including adaptation of existing point cloud generation task metrics, and the distance between momenta distributions for real and generated jets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The proposed application of point cloud generative models is novel and interesting. The authors promise to release the dataset to encourage researchers to contribute to the task.\n* Both qualitative and quantitative evaluations show that the proposed message passing generators and discriminators yield better performance compared to straightforward application of point cloud specific alternatives.\n\nCons:\n* The novelty of the approach is somewhat limited, since the overall proposed model is a combination of existing point cloud generative approaches and prior message passing networks. However, given the novel application context I do not think it is a significant drawback.\n* Related work section could be improved by incorporation of existing point cloud generative models based on probabilistic modeling [1, 2, 3]. These approaches naturally overcome the issues of models with fixed point cloud size outputs and, in general, might be of interest to the authors.\n\nThe overall quality of the submission is sufficient and even though the novelty is limited, I think it is an interesting enough application context to allow it to be published to draw attention to the task.\n\nQuestions:\n1) What was the number of message passing steps used in the considered architectures?\n2) Were there any ablations done for the configuration of the final generators and discriminators in the proposed MPGAN?\n3) You mention that one of the problems with application of ML-based generative models to physics in the lack of interpretability. You try to justify the use of such approaches if they would achieve the quality levels comparable to real baselines. Does these baselines give any guarantees on the objectivity of the resulting jet data? In case of GANs there are no such guaranties. Even if the mean metric aggregated over the whole evaluation dataset is relatively close to the aggregated metric calculated on the baseline data, there could be generated samples, which could be completely unrealistic (models are not strictly constrained to exclude them). If applied in the real-life scenarios, how generative approaches can overcome this issue?\n\n\n[1] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: PointFlow: 3D point cloud generation with continuous normalizing flows. In ICCV’19.\n\n[2] Klokov, R., Boyer, E., Verbeek, J.: Discrete Point Flow Networks for Efficient Point Cloud Generation. In ECCV’20.\n\n[3] Cai, R., Yang, G., Averbuch-Elor, H., Hao, Z., Belongie, S., Snavely, N., Hariharan, B.: Learning Gradient Fields for Shape Generation. In ECCV’20.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The impact and limitations were briefly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers really liked this paper. They especially noted the clarity of exposition and the fact that the authors intend on open sourcing their data. During the rebuttal phase, the authors added new experimental results using TreeGAN that helped to address some concerns of the reviewers. Overall this seems like a timely and interesting contribution to the literature on point clouds and GANs.
Decision: Accept (Poster)