SUPA: A Lightweight Diagnostic Simulator for Machine Learning in Particle Physics
Authors: Atul Kumar Sinha
Daniele Paliotta
Bálint Máté
John Andrew Raine
Tobias Golling
François Fleuret
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: HEP
Point Clouds
Generative Models
Simulation
Synthetic dataset
Calorimeter

Abstract: Deep learning methods have gained popularity in high energy physics for fast modeling of particle showers in detectors. Detailed simulation frameworks such as the gold standard \textsc{Geant4} are computationally intensive, and current deep generative architectures work on discretized, lower resolution versions of the detailed simulation. The development of models that work at higher spatial resolutions is currently hindered by the complexity of the full simulation data, and by the lack of simpler, more interpretable benchmarks. Our contribution is \textsc{SUPA}, the SUrrogate PArticle propagation simulator, an algorithm and software package for generating data by simulating simplified particle propagation, scattering and shower development in matter. The generation is extremely fast and easy to use compared to \textsc{Geant4}, but still exhibits the key characteristics and challenges of the detailed simulation. The proposed simulator generates thousands of particle showers per second on a desktop machine, a speed up of up to 6 orders of magnitudes over \textsc{Geant4}, and stores detailed geometric information about the shower propagation. \textsc{\textsc{SUPA}} provides much greater flexibility for setting initial conditions and defining multiple benchmarks for the development of models. Moreover, interpreting particle showers as point clouds creates a connection to geometric machine learning and provides challenging and fundamentally new datasets for the field.

SUPA: A Lightweight Diagnostic Simulator for Machine Learning in Particle Physics
Atul Kumar Sinha University of Geneva atul. sinha@unige.ch
Daniele Paliotta University of Geneva
Bálint Máté University of Geneva
daniele wwwiottauunige..commmmm ch
balint www.mmmmgeeee....m
John A. Raine
Tobias Golling
University of Geneva, CERN j johnny. raineecern.cc..o. ch
University of Geneva, CERN tobias gollingScern.co ch
François Fleuret University of Geneva francois www.urettunnig....mmm ch
Abstract
Deep learning methods have gained popularity in high energy physics for fast mod- eling of particle showers detectors. Detailed simulation frameworks such as the gold standard GEANT4 computationally intensive, and current deep generative architectures work on discretized, lower resolution versions of the detailed simula- tion. The development of models that work higher spatial resolutions is currently hindered by the complexity of the full simulation data, and by the lack of simpler, more interpretable benchmarks. Our contribution is SUPA, the SUrrogate PArticle propagation simulator, an algorithm and software package for generating data by simulating simplified particle propagation, scattering and shower development in matter. The generation is extremely fast and easy to use compared to GEANT4, but still exhibits the key characteristics and challenges of the detailed simulation. The proposed simulator generates thousands of particle showers per second on desktop machine, speed up of up orders of magnitudes over GEANT4, and stores detailed geometric information about the shower propagation. SUPA provides much greater flexibility for setting initial conditions and defining multi- ple benchmarks for the development of models. Moreover, interpreting particle showers as point clouds creates a connection to geometric machine learning and provides challenging and fundamentally new datasets for the field.
Introduction
In order to understand the fundamental building blocks of nature, High Energy Physics (HEP) experiments involve highly energetic partiele collisions. These collisions cause particles decay, and the identification the resulting decay particles of key importance to develop and confirm new physics/theories. This is enabled through electromagnetic or hadronic calorimeter. As particles interact with the calorimeter, they split into multiple other particles, forming a particle shower. The nature of the generated shower depends the specific material and geometry of the calorimeter, which are carefully selected and driven by physics theory and pratical considerations. The generated shower deposits energy in active layers of the calorimeters, and provides energy and location measurements from the particles produced in the cascade.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Precise computer simulation is central to a better understanding of experimental results in HEP. Simulation is especially useful for the task of event reconstruction, where the deposited energy in the calorimeter is used to identify the particle that originated given shower. The pattern of the deposited energy depends on the specific type of the (intermediate) particles, the initial energy and incidence angle, as well as the specific geometry and shape of the detector material, among other factors. The state-of-the-art for this kind of simulations is GEANT4 [Agostinelli et al. 2003). a Monte Carlo toolkit for modeling the propagation of particles through matter.
While detailed simulation through GEANT4 provides fine-grate grained shower generation and captures the underlying distributions accurately, this software consists of more than 3.5 million lines of C++, is computationally very expensive, and requires specific domain knowledge to be set up and tuned. This makes it cumbersome and expensive to produce the amount and diversity of data needed to speed up machine learning research on these topics. Table compares GEANT4 with our proposed simulator SUPA.
Our key contributions ease these issues are:
We introduce SUPA:
fast, easy to use simulator for simple particle propagation, scattering
and shower development in matter (see $ $3.
We structure the simulator in a way that allows to easily change the complexity of the under- lying shower development as well as the properties of the showers in terms of multiplicity.
energy range, structure (see $ $4.1I.
We use SUPA to generate data that highlights the limitations of current machine learning approaches for fast shower simulation (see $ (3.2). We experimentally demonstrate that SUPA can be used as proxy for detailed simulation, while being easy to tune and up to orders of magnitude faster than GEANT4.
1.1 Image Representation
Existing approaches for generating particle showers (Paganini et al. 2018 Krause and Shih 2021 focus quantized representation of the calorimeter hits, binning them into discrete pixels, or cells, and setting the pixel intensity to the sum of the energy deposited. The upside of this approach is that, due to the structure of existing calorimeters, the data takes the form of low resolution images and standard computer vision models can be directly applied. However, as we increase the resolution of the quantized representations in order preserve more information, the resulting images become exceedingly sparse, leading to difficulties in training existing machine learning architectures. Figure 1b Ic and [d] show the downsampled image representations resolutions of 3x, 2x and 1x respectively for the shower shown in Figure Ia We choose 1x to be the same resolution as used in CaloGAN PPaganini al. 2018 (i.e. 12 12 for Layer 1).
is also important to notice that, when decreasing the resolution the data and looking at showers, while the signal to model is of lower dimension and less sparse, the underlying structure and rela- tionships between hits are lost. This requires models to implicitly learn equivariances from multiple examples, while also introducing artificial structure arising from the chosen detector geometry. For example, with a shower incident on the surface of a detector, a slight translation in either direction does not change the properties of the shower. However it still has impact on the resulting datasets, where for small shifts it is not translationally equivariant. Having access to the underlying energy depositions would therefore greatly reduce the dependence on the geometry of the detectors in the models, and allow generative models to learn the underlying structure from the physical process directly.
1.2 Point Cloud Representation
Many popular point-cloud datasets Wu et al., 2015 Chang et al. 2015 in the machine learning community contain shape representations of physical objects, such as points sampled uniformly from the surfaces of chairs, tables, etc. These datasets enjoy many nice properties, e.g. locality and redundancy, that calorimetry datasets do not. Moreover, existing generative models for point clouds rely on these simplifying assumptions and therefore their performance on more diverse datasets is still relatively unexplored. In the case of particle showers in calorimeters, dealing directly with the point cloud representation provides an optimal description of the underlying physics processes. However,

---

(a) Point Cloud
(b)
(c) 2x
(d)
Figure 1: An example shower (single layer) shown at different resolutions
due to the detailed simulation resulting from GEANT4, the number of points per shower represents huge computational challenge.
key objective of our synthetic simulator is to bridge this gap by allowing for quicker ground truth data generation for different initial settings, thus providing way of modulating the complexity of the dataset, leading to more interpretable benchmarks. This would also make it possible to take gradual steps in developing the models and work which of the many complexities the GEANT4 datasets are hardest to model.
2 Related Work
Shower simulation with GEANT4 requires a complete description the geometry and material of the detector and simulates all interactions between the initial particle, any subsequently produced particles through decays or emissions, and the detector material. The upside of using GEANT4 for simulation is that it is very accurate and represents the underlying physical phenomena precisely, but it comes at the cost of being computationally very expensive. Moreover, requires expertise to define the configuration and multitude of expert hours set it up for detector geometries. To reduce the computational resources required to simulate particle physics collisions and their energy depositions in detectors, deep generative models have been applied generate particle showers. Most of these models [Abhishek et al. 2021 Krause and Shih 2021 Paganini et al. 2018 employ the dataset first introduced in CaloGAN Paganini al. 2018 which based on simplified calorimeter inspired by the ATLAS Liquid Argon (LAr) electromagnetic calorimeter,
2.1 Datasets and calorimeter structure
For the CaloGAN dataset, the calorimeter is cubic in shape with each dimension being 480 mm and no material in front of it. The volume is divided into three layers along the radial (z) direction with varying thicknesses of 90 mm, 347 mm, and 43 mm, respectively. These layers are further segmented into discrete cells, with different sizes for each layer: 160 mm x 5 mm (first layer), 40 mm x 40 mm (second layer) and mm x 80 mm (third layer). Each layer can be represented as a single-channel two-dimensional image with pixel intensities representing the energy deposited in the region. The final read-out has the resolution of 3 x 96, 12, and 12 x For this dataset, three different particle types were considered, namely, positrons, charged pions and photons. Further, the particles were configured to be incident perpendicular to the calorimeter with initial energies uniformly distributed in the range between GeV and 100 GeV.
Several other calorimeter geometries and configurations are employed across the literature. Erdmann et al. 2019) consider calorimeter configuration motivated by the CMS High Granularity Calorimeter (HGCAL) prototype, while Belayneh et al.) (220) study calorimeter based on the geometric layout of the proposed Linear Collider Detector (L (LCD) for the CLIC accelerator, Buhhmann et al.) (2021b investigate the prototype calorimeter for the International I. Large Detector (II.) which one of two proposed detector concepts for the International Linear Collider (II.C). The Fast Calorimeter
Simulation Challenge 2022 (CaloChallengee0222) Faucci Giannelli et al.) [002cc a b was proposed to spur the development and benchmarking of fast and high-fidelity calorimeter shower generation using deep learning methods. The challenge proposed three datasets, ranging in difficulty from easy
'The ATLAS calorimeter has e.g. a more complex geometry, with the cells having accordion shaped electrodes to maximise the active volume
3

---

to medium to hard. the difficulty is determined by the dimensionality of the calorimeter showers i.c. number of layers and number of voxels in each layer. In different application setting, |Erdmann et al.] (2018) model calorimeter response for cosmic ray-induced air showers in the Earth's s atmosphere, producing signals in ground based detector stations. In contrast to the CaloGAN calorimeter, it has a single readout layer with 9 x 9 cells, each cell corresponds to detector unit placed with a spacing of 1500 m. Although many different detector technologies are considered in the various models, the common aspect between the models and the corresponding datasets is the projection of the spatial signal onto discretized cells in 2D planes or a generalized 3D volume. The projection is done either directly while designing the calorimeter geometry, as a post processing step. Further, the number of cells and thus the resolution of the final read-out is usually kept small. This is done either to simplify the data for training the models, or in order to speed up the simulations.
Due to the different geometries used for generating the datasets, compare the performance of the various models. As the developments of each shower are completely dependent on the detailed model of the detector in GEANT4, although the same initial particles can be simulated, it is very difficult to draw parallels between two different datasets, as it is not possible simply to change the representation of the data from one model to look like that of another.
not a straightforward task to
2.2 Deep Generative Models
The applications of deep generative modeling to calorimeter simulation have far almost entirely focused on Generative Adversarial Networks (GANs, Goodfellow et al. 2014) and Normalizing Flows [Rezende and Mohamed 2016). CaloGAN (Paganini al. 2018 was the first application of GANs to a longitudinally segmented calorimeter. The approach is based on LAGAN (de Oliveira et al., 2017]. a DCGAN [Radford et al. 2016 -like architecture, that is able synthetize the shower images. The generator outputs a gray-scale image for each layer in the calorimeter, with each output pixel representing the energy pattern collected at that location. The architecture is also complemented with an auxiliary classifier tasked with reconstructing the initial energy E.
Krause and Shih [202T] improve on CaloGAN with CaloFlow, normalizing flow architecture to generate shower images. CaloFlow is able to generate better samples than previous approaches based on GANs and VAEs, while also providing more stable training. Inductive CaloFlow (iCaloFlow)
Buckley et al. [2023] extends the CaloFlow model to higher granularity detector geometries. It utilizes teacher student distillation to increase sampling speed without loss of expressivity.
Previous work by ATLAS collaboration [2018) proposes to use Variational Autoencoder on the flattened images [Kingma and Welling [014] Both the encoder and decoder consist of an MLP
conditioned on the energy of the initial particle. In addition to the reconstruction and KL.-terms, the authors also optimize for the overall energy deposit in both in the individual layers and in the overall system.
Buhmann et al. 2023) proposes a diffusion-based generative model representing one of the initial methods devised for effectively processing the point cloud representation of calorimeter showers.
3 Propagation Model
SUPA generates data that imitates the propagation and splitiigg/saattringg of point particles. To generate an event, a point particle is initialized at the origin of the 3D space with an initial energy value and velocity parallel to the z-axis. We then define sequence of 2D planes orthogonal to the z-axis at [=o...., =N] that we call "slices" (or "sub-layers", interchangeably) We assume that the slices are evenly spaced along the z-axis, with a gap of A between consecutive slices. As the initial particle reaches the first predefined z-slice at 2g. it has 3 options (see Figure stop and deposit its energy with stopping probability Pstop- split into two with splitting probability Paplit- simply continue moving with the same velocity with pass-through probability Ppass = 1 - (Pstop + Psplit). Then:
* If the particle splits, two particles are generated at the splitting location with energy and velocity features that ensure momentum conservation, i.e., that the sum of momentum
A continuous attribute obeying simple conservation laws at each splitting, which can be interpreted as an equivalent for the energy of the particles in shower. As such, we refer to this property the energy.

---

Lape11
Layer
L
Pypl
Pacp
Pyess
Figure 2: Schematic summary of a propagation event generated by SUPA. (left) Splitting events happen at the black vertical lines, called slices. See $ 3 for detailed explanation of the process. (right) Each time particle crosses pre-defined "slice", it either splits in two, stops and deposits energy, or passes through unperturbed.
vectors the new particles equal to the decaying onc. The original particle is removed from the propagation model and the two ones continue propagating with their respective velocities. The splitting angle 0 and deviation angle are sampled from distributions pe (.) and P.)., respectively.
If the particle stops, its energy is deposited and the energy value and the location of the hit are recorded.
If the particle passes through, it simply continues with its original velocity.
See Algorithm]r for the pseudocode of this process. We can think of the process detector layer being positioned at the sg-slice recording energy deposits. After the slice at žg, the model either contains 0,1 or particles still propagating. As they reach the additional slices at {z1,..., Z the above procedure is repeated: whenever a particle is at a z-slice, it either passes, deposits its energy or decays.
After all particles crossed the last slice at nn we collect all energy deposits (energy value and 3D location). Instead of building dataset from the exact 3D location of these deposits, we partition the slices into subsets we call layers. We refer to the partitioning scheme layer configuration.
We then build an event from the energy deposit L' and the (x, y) yo-oonnnee of all hits for all slices within a given layer Li, effectively projecting the z-axis into single point. An event then consists of N 2D point clouds, one for each layer, with a single scalar feature per point. Intuitively, the slices z; are the positions where the particle can split and stop (granularity the simulation). and the layers L; correspond to the detector layers, where the readout happens. Figure Segureeccom contains a schematic summary for the propagation process.
Having complete control over the algorithm, we can increase/decrease the complexity of the generated data by adjusting the distribution of the properties of the initial particle, the number and location of the slices 2,, the probabilities Pstop=Ppase Paplit and the distribution of the properties of the children after a splitting event. This especially useful debug and understand the behaviour of machine learning models at different levels for complexity.
Table 1: Comparison of GEANT4 and the proposed model (SUPA)
GEANT4
SUPA
3.6m lines in C++
few hundred lines
codebase
in Python
detailed physics
fast data
target usage
simulations
generation
speed
1772 ms/showe
0.1-100 ms/showe
5

---

Algorithm SUPA
Particle Splitting
Parameters:
Stopping probability; Pstop Splitting probability: Peplis Layer configuration
2: 0 po(.), pe(.) U(001)
3: if Pstop [ then
Active t- False
L 20 Zig
5: else if Patop [4] < a < Putop + Papli then
L2 = 21g 11 25g
6: w <- (0, 2xx)
...
7: Po p+ Ao (sin c), cos(0/2 e))
L, {
2NJ.
Slice Gap: A
8: P1 - p A: (sin /8/ c), cos +))
Splitring angle: po (.)
Deviation angle: (c(.)
9:
E(o.1) E (0/2 1 e)/0
Input:
Tree T, Position p (An) AΦ),
10: T.add( 1o). add pp) , 11)
Energy E. Layer
11: else
Output:
collection of {px, F
12: T.add(p, | 1,
of generated particles
13: end if
1: E,,
Input, N p(8(m 2))
14: Return T
Notation. We denote single shower {XL, XL2 XLx where XL, denotes the 7t layer and N is the total number of layers considered. For the point cloud representation, XL, can further where N; denotes the total number of points in layer of the
expanded XL, x2...!...... x,, shower (can vary across showers). Further, each point is vector x [ni, G3. Ej]. where r and are the coordinates and E denotes the energy.
Using SUPA
SUPA provides various meta-parameters which affect the data generation and various characteristics of the generated data. This allows to generate shower data with varying levels of complexity and thus enables incremental benchmarking. For instance, Psplit affects the number of emerging secondary particles and thus implicitly the number of hits (or points), while Pstop controls the position (in depth) of the observed hits. controls the overall spread of the hits in the lateral plane and a controls the dynamic range of the observed energy values. The different configurations for these parameters is analogous to different detector geometty/matcrial or perhaps a different particle.
4.1 SUPA Datasets
We propose five standard configurations and the resulting datasets SUPAv1-5 for our analysis and benchmarking, summarized in Table All configurations have the same initial energy of 65 GeV, initial angle of π/ and initial impact position (0, 0), i.e. particles are incident perpendicular and at the center of the calorimeter. Section A.2 the appendix gives more details and also shows the interpretation of these variations with respect to different summary variables. SUPAv1 is deterministic
Table 2: Summary of different datasets generated with SUPA
Dataset
a
Psplits Pstops Poass
Points
Layer Configuration
SUPAw1
0
see Fig.
5a
128
L0: [5, 10
SUPAv2
0
see Fig.
5b
1,91
L0: [7, 20
74
SUPAv3
U
21 0
see Fig.
5b
1,91
L0: [7,20 20
32: 16
SUPAv4
U(
5
21
0
0
see Fig.
5b
1, 1484
L.
[7, 12
32
16
SUPAv5
u( 371
U1
8
see Fig.
5c
5, 280
LO: [7, 12). LI [13, 28
'The exact simulation time depends on the incident particle and its kinematic properties. as well as the detailed composition of the detector geometry in the GEANT4 model. These numbers taken for the CaloGAN geometry introduced Paganini et al. (2018).
'Reflects the time to generate directly the point cloud representation, which is more granular than the corresponding GEANT4 generation

---

with respect to splitting and stopping probabilities as well as in & and a. It has the same number of points across all events and all the hits have the same energy value. SUPAv2-4 have the same configuration for splitting and stopping probabilities (see Sec. A.2.Ip, while SUPAv2 is deterministic in 0 and a. SUPAv3 is naturally more spread out than SUPAv2 (see Fig. 6b vs. Fig- 66 and also distributions for 7) d in Fig. 7 and (1)) (o), (r) in Fig, 8). SUPAv4 is similar to SUPAv3, except for the layer configuration. SUPAv4 only has a subset of the sub-layers by dropping the last sub-layers from SUPAv3. While SUPAv3 captures the total initial energy in Layer 0, SUPAv4 has some energy leakage which is apparent from the distribution of layer energy in Fig. 10f Another interesting structure in shower data is the dynamic range of the energy values in given shower. A high dynamic range of energy makes learning generative models difficult [Krause and Shih 2021]. With SUPA, we can control the dynamic range via the parameter SUPAv1 and SUPA22 have fixed o-= 0), thus all the splits are symmetrie leading to a low dynamic range (see Fig. 10d for histogram of variance in energy values). The other factor which affects the dynamic range is the layer configuration, more number of slices in a layer would lead to more dynamic range. SUPAY5 has multiple layers and has higher Psplit in the initial slices, thus it is more complex in terms of multiplicity or total number of hits. Please refer to the appendix Sec. A.2 for more details and other canonical SUPA datasets.
4.2 Extensions
SUPA very flexible and can be easily extended. The slices can be further divided into different regions each with their own meta weta-parameters providing more granular control over the scattering and splitting process. Another variation requires having some slices that are not merged into layers to mimic dead material. These extensions can possibly bring it closer to more realistic detectors, and lead to unique structures in the dataset and thus present challenges for model development and evaluation. It is also possible condition the data generation on various attributes, for example: initial impact angle, impact energy, impact position, etc. This encourages interesting studies and benchmarks regarding extrapolation and generalization capabilities of generative models for unseen conditioning variables during training.
5 Evaluation and Analysis
The evaluation of generative models for high dimensional data is a non-trivial problem. While reconstruction losses and/or data likelihood are available for some model architectures (e.g. auto- encoders or flows), judging the sample quality still difficult. Moreover, the average lig-likelihood is difficult to evaluate or even approximate for many interesting models. For instance, in computer vision, evaluating generative models for images is done via proxy measures such as the Frechet Inception Distances (FID, |Heseel et al.) 2017). Point cloud generative models are evaluated (Yang) (FID.Heesee
et al. 2019) using metrics such as Minimum matching distance (MMD), Coverage (COV) and 1-Nearest Neighbour Accuracy (1 (I-NNA), where similarity between a set of reference point clouds and generated point clouds are measured using a distance metric such as the Chamfer distance (CD) or Earth mover's 's distance (EMD) based on optimal matching.
5.1 Shower Shape Variables
For calorimeter simulation, we cannot judge performance using individual examples due to the stochastic nature of shower development. The standard practice to estimate the quality of generative models for calorimetry data (shower simulations) is to use histograms of different metrics called shower shape variables [Paganini et al. [0188) in addition to images of calorimeter showers. These shower shape variables are Nomain--specin and physically motivated, and matching densities over these marginals is indicative of matching salient aspects of the data. These marginals capture various aspects of how energy is distributed within individual layers and across different layers.
We extend the shower shape variables for the point clouds representation of showers. A summary of all the shower shape variables considered are present in the appendix.
Point level marginals. Marginals of each point feature by considering the set all the points from all the point clouds together.
7

---

Feature means (ni), (0.), (rr)) (Et). Mean of each feature.
(m:) = Σ3 m3 Σ, (0.) Σ, of Σ3
Σ
Σ,
where rj / (nr))2 + (ot)2 denotes the distance of the point in the lateral plane from the center.
E n - (mx)
Feature variances (m.) (b)): (nn) (E))) Variance of each feature. ((x) = V Σ,
Layer Energy E,. Denotes the total energy deposited in layer i of the shower. Ei = EjEN, Ej.
Total Energy Eus- Total energy across all layers of the shower. E Eisn E..
Layer Centroids (16) E, (di) (ri) Energy weighted mean of the features (n, d. or
(m)) = Σ, Flgn (o.) = Σ, Ege (r)) Σ, Rjr E,
E,
E,
The layer centroids can be interpreted as the center of energy in the lateral plane in respective dimensions. Layer Lateral Width o (ndg (b)) (r)) Denotes the standard deviation of the layer centroids.
Σ, E )n E,
(n:)
(m))
The layer lateral widths can be interpreted as the spread around the center of energy in the lateral plane in respective dimensions. We drop the layer notation i from the above metries when working with single layer for brevity.
5.2 Performance Analysis of Generative Models
In order to summarise the discrepancy into scalar value, we consider the Wasserstein-1 I distance between the histograms of the ground truth and generated sample's 's marginal Further, we compute the mean discrepancy over different groups of marginals summarize the performance of models.
We train point cloud generative models, PointFlow (Yang al. 2019], SetVAE [Kim al. 2021], and a transformer-baat Table summarrizs the performance of different models across different variations of SUPA datasets. Pointilow performs better than SetVAE for all marginals not involving energy feature such as point level marginals, feature means and variances for location parameters, etc., and worse for marginals which depend on energy such as, layer centroids, and layer energy, etc. (with the exception of layer lateral widths and feature variance (E)), which are both second order moments), across all datasets. Roughly speaking, PointFlow struggles at modeling the energy feature while it is competitive for the location features, and SetVAE behaves opposite. It is interesting to observe that Transflowmer is either the best performing very close to best performing model (either SetVAE or PointFlow) across all datasets and marginals, indicating relatively greater flexiblity at modeling a variety of features. The instances with relatively high values in Table 3 (feature means for SUPAv1, and layer energy for SUPAv2 and SUPAv3) are where the marginals had a very narrow spread, indicating the difficulty in modeling discrete distributions. The models also have the tendency to predict points with very high energy values unseen during training (see Fig. 24a). We provide more details in the Appendix Section A.4 the training, results and analysis, along with plots of various shower shape variables to visually compare the goodness of fit.
flow model (which we call Transflowmer, see Sec. A.3). on SUPA data.
These observations highlight the need for adapting point cloud generative models such that they are able to model points with special features (different from location parameters), such as physical features like energy, momentum, etc. Although the location parameters are correlated with these special features, the overall structure of this correlation can be very different from domain to domain and also from how they correlated with other location features.
For computing the Wasserstein distance, we normalize the shower shape variables for the ground truth and the generated sample according to the mean and standard deviation of the ground truth.
8

---

Table 3: Performance benchmarks across different datasets with SetVAE, PointFlow and Trans- flowmer. The distance metric is Wasserstein- 1. The reported numbers are averages over group of marginals as indicated in the top row. Lower numbers are better. See $ 5.1 for more details on the
metrics.
Point Features
V), Φ,
Point Feature
E
(m)), (0.), (rs)
Dataset
SV
PF
TF
SV
PF
TF
SV
PF
TF
SUPAv1
0.044
0.037
0.028
0.001
0.000
0.000
16.254
21.921
5. 5.789
SUPAv2
0.290
0.167
0.029
0.253
0.462
0.145
0.517
0.256
0.091
SUPAv3
0.510
0.130
0.044
0.304
0.484
0.048
0.769
0.075
0,105
SUPAv4
0.269
0.122
0.029
0.111
0.459
0.006
0.460
0.094
0.073
SUPAvS
0.166
0.028
0.042
0.078
0.090
0.151
0.592
0.031
0.189
(mn) 0 (d2) ((r)
(b)
((E)
Dataset
SV
TF
SV
PF
TF
SV
PF
TF
SUPAv1
0.513
0.585
0.359
0.001
0.000
0.000
0.000
0.000
0.000
SUPAv2
0.648
0.154
0.130
0.302
0.077
0.087
0.513
0.177
0.165
SUPAw3
1.114
0.109
0.126
0.320
0.064
0.071
0.500
0.152
0.144
SUPAv4
0.634
0.092
0.051
0.263
0.047
0.022
0.355
0.156
0.038
SUPAv5
0.799
0.040
0.223
0.251
0.059
0.414
0.377
0.047
0.421
(m)). (oi) g. (rv)
0 (mc) R ((c) (rr))
E
Dataset
TF
SV
PF
TF
SUPAv1
16.244
21.921
5.766
0.517
0.591
0.379
0.101
0.000
0.039
SUPAv2
1.336
1.933
0.137
0.779
0.134
0.190
23.387
69.814
10.468
SUPAv3
1.365
1.627
0.217
1.226
0.147
0.151
36.116
79.062
5.916
SUPA
1.369
1.346
0.083
0.645
0.169
0.067
3.997
12.895
0.659
SUPAv5
2.373
1.615
0.463
0.740
0.058
0.317
6.132
16.742
9.273
5.3
SUPA as a benchmark
We train the models of $ 2222 on both the events generated by GEANT4 and SUPA. For these studies, we generated another version of the dataset with SUPA and downsample the point clouds to grid
representation such that is similar to the CALOGAN dataset (see A.5 for details on data generation). The training loss of the flow model is log-likelihood which meaningfully represents the quality of the captured distribution. of different variations of the CaloFlow architecture trained both GEANT4 and -genrrrede events. Figure shows the scatter plot of the mean discrepancy (across all marginals) obtained by different models on GEANT4 and SUPA- SUPA-generated events. We show more plots in the Appendix $ A..
displays the correlation of the log- likelihood values
100.0
Caleflow
20.0
CaleGAN
0.0
100.0
5/PA SLL NLL
200.0
400.0
.30.0
600.0
-100.0
200.6
4n0.0
geant
Figure 3: Scatter plot for loss over different ver- sions of CaloFlow models. Each point corre- sponds to an architecture, its x and coordinates are the negative oogliiklihoods on data gener- ated by GEANT4 and SUPA, respectively. Point size reflects the capacity of the model (number of layers, learnable parameters, etc.).
Figure 4: Scatter Plot for mean discrepancy ($ over different models. Lower numbers better. Performance of models are consistent over both datasets, better model on SUPA implies bet- ter model on GEANT4 and vice versa.

---

We can observe that all these correlation plots are roughly monotonic, showing that benchmarking on SUPA provides good estimate of relative performance on the detailed GEANT4.
6 Conclusion
We introduced SUPA, a lightweight pseudo-partice simulator that is inspired by the physics governed development of particle showers and qualitatively resembles the data generated by the gold standard GEANT4. By allowing to easily change the underlying parameters of the propagation model, we introduce the freedom define new benchmark datasets of varying complexity with a fidelity and simplicity not available using the GEANT4 toolkit.
We believe that the ease of use, flexibility and speed of this simulator could be a MNIST moment for the field, allowing the rapid exploration of models various regime of functioning as discussed in $ We also highlight the gaps in the ability of existing point cloud generative models on calorimetry data in $ 5.2 and highlight the need for more powerful point cloud generative models.
Additionally, we showed in 5.3 and $ A.5.1 that with grid representation of the shower data, performance of deep generative models estimated on data produced with this simulator a good proxy for their performance on the standard data used historically in machine learning for particle physics. We highlight flexibility of SUPA in $ A.5.2 allowing us to train/evaluate at different resolutions.
Moreover, by giving easy access to the fully recorded event, we open the way to applying methods from point clouds and geometric deep learning research, with the goal of possibly devising end-to-enu architectures able reconstruct a full event from the energy deposits by solving the complete inverse problem, something currently impossible with data from GEANT4.
References
Abhishek Abhishek, Eric Drechsler, Wojeiech Fedorko, and Bernd Stelzer. CaloDVAE: Discrete variational autoencoders for fast calorimeter shower simulation. 2021.
S. Agostinelli, J. Allison, K. Amako, J. Apostolakis, H. Araujo, P. Arce, M. Asai, D. Axen, S. Banerjee, G. Barrand, F. Behner, L. Bellagamba, J. Boudreau, L. Broglia, A. Brunengo, H. Burkhardt, S. Chauvie, J. Chuma, R. Chytracek, G. Cooperman, G. Cosmo, P. Degtyarenko, A. Dell' Acqua, G. Depaola, D. Dietrich, R. Enami, A. Feliciello, C. Ferguson, H. Fesefeldt, G. Folger. F. Foppiano. A. Forti, Garelli, S. Giani, R. Giannitrapani, D. Gibin, J.J. Gómez Cadenas, I. González. G. Gracia Abril, G. Greeniaus, W. Greiner, V. Grichine, A. Grossheim, S. Guatelli. P. Gumplinger, R. Hamatsu, K. Hashimoto, H. Hasui, A. Heikkinen, A. Howard, V. Ivanchenko, A. Johnson, F.W. Jones, Kallenbach, N. Kanaya, M. Kawabata, Y. Kawabata, M. Kawaguti, S. Kelner, P. Kent, A. Kimura, T. Kodama, R. Kokoulin, M. Kossov, H. Kurashige, E. Lamanna, T. Lampén, V. Lara, V. Lefebure, F. Lei, M. Liendl, W. Lockman, F. Longo, S. Magni, M. Maire, E. Medernach, K. Minamimoto, P. Mora de Freitas, Y. Morita, K. Murakami, M. Nagamatu, R. Nartallo, P. Nieminen, T. Nishimura, K. Ohtsubo, M. Okamura, O'Neale, Y. Oohata, K. Pacch, J. Perl, A. Pfeiffer, M.G. Pia, F. Ranjard. A. Rybin, S. Sadilov. E. Di Salvo, G. Santin, T. Sasaki, N. Savvas, Y. Sawada, S. Scherer. S. Sei. V. Sirotenko, D. Smith, N. Starkov, H. Stoecker, J. Sulkimo, M. Takahata, S. Tanaka, E. Tcherniaev, E. Safai Tehrani, M. Tropeano, P. Truscott, H. Uno, Urban, P. Urban, M. Verderi, A. Walkden, W. Wander, H. Weber, J.P. Wellisch, T. Wenaus, D.C. Williams, D. Wright, T. Yamada, H. Yoshida, and D. Zschiesche. Geant4 a simulation toolkit. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 506(3): 250-303, 2003. ISSN 0168-9002. doi: http:://bb.orrg11000066600666660 (000)///S0000666000000001333333 URL
ATLAS collaboration. Deepaioonnnnnnneeeeeeeee generative modeiiiieemmdleeeee models for fast shower siwulation in atlas. http://cdsweb
cern. http:/a/eeaaeeee//.////aaa//aaaa 01-26.
2018. Accessed: 2022-
Dawit Belayneh, Federico Carminati, Amir Farbin, Benjamin Hooberman, Gulrukh Khattak, Miaoyuan Liu, Junze Liu, Dominick Olivito, Vitória Barin Pacela, Maurizio Pierini, et al. Calorime-
10

---

try with deep learning: particle simulation and reconstruction for collider physics. The European Physical Journal C, 80(7):1-11, 2020.
Matthew R Buckley, Claudius Krause, Ian Pang, and David Shih. Inductive caloflow. arXiv preprint arXiv:2305. 11934, 2023.
Erik Buhmann, Sascha Diefenbacher, Engin Eren, Frank Gaede, Gregor Kasieczka, Anatolii Korol, and Katja Krüger. Decoding photons: Physics in the latent space of a bib-ae generative network. arXiv preprint arXiv:2102.12491, 2021a.
Erik Buhmann, Sascha Diefenbacher, Engin Eren, Frank Gaede, Gregor Kasieczka, Anatolii Korol, and Katja Krüger. Getting high: high fidelity simulation of high granularity calorimeters with high speed. Computing and Software for Big Science, 5(1):1-17, 2021b.
Erik Buhmann, Sascha Diefenbacher, Engin Eren, Frank Gaede, Gregor Kasieczka, Anatolii Korol, William Korcari, Katja Krüger, and Peter McKeown. Caloclouds: Fast geometry- -ineeery-nndeeendent highly-granular calorimeter simulation. arXiv preprint arXiv:2305 00847,, 2023.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv: 1512.0012 [cs.GR], Stanford University Princeton University - Toyota Technological Institute at Chicago, 2015.
Luke de Oliveira, Michela Paganini, and Benjamin Nachman. Learning particle physies by example: Location-aware generative adversarial networks for physics synthesis. Computing and Software for Big Science, 1(1) Sep 2017. ISSN 2510-2044. doi: 10.1007/441781-007-00006 URL htt.::/.oo..oogoo////oooo 0tt.:/.....00..000///10000000000
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. CoRR, abs/1605.08803, 2016. URL http:://aaxi...rg/ab////b///////
Martin Erdmann, Lukas Geiger, Jonas Glombitza, and David Schmidt. Generating and refining particle detector simulations using the wasserstein distance in adversarial networks. Computing and Software for Big Science, 2(1): 2018.
Martin Erdmann, Jonas Glombitza, and Thorben Quast. Precise simulation of electromagnetic calorimeter showers using wasserstein generative adversarial network. Computing and Software for Big Science, 3(1):1-13, 2019.
Michele Faucci Giannelli, Gregor Kasieczka, Claudius Krause, Ben Nachman, Dalila Salamani, David Shih, and Anna Zaborowska. Fast Calorimeter Simulation Challenge 2022 Dataset 2,
March 2022a. URL https://doi. wtt.:/org//........ 520..022222/........ 136622
Michele Faucci Giannelli, Gregor Kasieczka, Claudius Krause, Ben Nachman, Dalila Salamani, David Shih, and Anna Zaborowska. Fast Calorimeter Simulation Challenge 2022 Dataset March 2022b. Utthttss://doi 1052///ddd/// 6366324
Michele Faucci Giannelli, Gregor Kasieczka, Claudius Krause, Ben Nachman, Dalila Salamani, David Shih, and Anna Zaborowska. Fast Calorimeter Simulation Challenge 2022 Dataset 1, March 2022c. URL https://doi. org/ 1t////10.5281//1 www.522811eeoooo 6368338
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q Weinberger, editors, Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL Paper pdf
neur cc/
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, 2017.

---

Jinwoo Kim, Jachoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composi- tion for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15059-15068, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Claudius Krause and David Shih. Caloflow: Fast and accurate generation of calorimeter showers with normalizing flows, 2021.
Michela Paganini, Luke de Oliveira, and Benjamin Nachman. Calogan: Simulating 3d high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks. Physical Review D, 97(1):014021, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1 2017. URL
http:///rxti http://axxvv.ooo www.:.rggggba..... httg//bb/1706..... 170..00000000 03762
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong
Xiao. 3d shapenets: A deep representation for volumetric shapes, 2015.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541 -4550, 2019.
12

---

A Appendix
A.1 Shower shape variables
We extend the list of shower shape variables described in Sec. 5.1
Layer Energy Fraction fi. Fraction of the total energy deposited in layer of the shower, ft
E Ema-
Energy Ratio i Ratio of the difference between highest and second highest energy intensity
Fij Fig
point or cell in layer i and their difference. Eratio, B +
Depth d. Deepest layer in the shower with non-zero energy deposit. max; (i: max, (E)) > 0}.
Layer/Depth Weighted Total Energy la. Sum of the layer energies weighted by the layer number. EigN E..
Shower Depth sa. Depth weighted total energy normalized by the total energy in the shower. $d = La/EE
Shower Depth Width de.- Standard deviation of sy in units of layer number.
Ei=n Es
Eizo Es
Exot
Exot
Brightest Voxels E; sei__riggtest__aeeee The 1eth brightest voxel in layer i normalized by the total layer energy. Eg_ E=_brightest__ayer = E.
Layer Sparsity. The ratio of the number of cells with non-zero energy to the total number of cells in layer i. This is only valid for the image representation of showers.
A.2
Details on different variations of SUPA datasets
A.2.1
Parameters
Fig. shows the remaining parameters used for generating SUPA variations (see Table for details on other parameters). SUPAv 1 is most deterministic as particles always split in the first six sub-layers with no deposits (Paplit = 1 and Pstop
for all sub-layers < 7), further since Pstop 1 sub-layer 7. all the particles get deposited. Thus each event/example in SUPAv1 has exactly 128(= 2²) points. Further, since a is fixed to 0, all splits are symmetric and energy is always halved at each split, thus all deposits have the same energy value. SUPAv5 has higher Psplit in the initial sub-layers (< 7) than
SUPAv2-4, while Pstop is the same for all of them, thus SUPAv5 has more number of hits/points than SUPAv2-4 in the respective sub-layers layers.
22-
(a) SUPAv1
(b) SUPAv2, SUPA03, SUPAv4
(c) SUPAv5
Figure 5: Parameters Puplits Pstops Ppass for SUPA variations
A.2.2 Shower Shape Variables
Fig- 6 shows the average events for different variations of SUPA datasets and Figs. Fig. 7 the histograms of the various shower shape variables for all SUPA datasets.
T2 shows
13

---

10.0
7.5
10
75-
301
5.0
Energy (MeV)
25
101
25
10
255
207
0.0
80
10
0.0
-25
25
10
25
550
3.
19a
59
107
10
10-2
-100
10-4
-150
10
-100
(a) SUPAv1
(b) SUPAv2
(C) SUPAY3
300
10.
75
101
301
50
5.0
25
107
2.5
20'
00
0.0
25
25
10a
5.0
-75
10-2
-10.0
10-2
(d) SUPAv4
(c) SUPAv5
Figure 6: Average event representation for different variations of SUPA datasets
10
10r4
SUnwa
10
10
10-2
SUNA4 SUNeS
bendty
1D-3
SU/v/
10-1
10-1
10-5
10
Figure Histograms of point level distributions
SUNA SUatk
10
Mikoul
Dendty
10.2
10-3
662
Figure 8: Histograms of feature means
10a
10a
SiNod SUROUS
SUNA1 SUsoul
10t
Derato
10
3
Ox
0H
Figure 9: Histograms of feature variances
14

---

10a
10
SING
10h1
SARNO
10.4.
Danaite
10--
20-4.
10-1
10-1
50
130
200
750
(a) Point dist.
Energy feature
(b) Number Points
(c) Energy Mean
201
10:
30%
Dordity
30-3
10
10-
(d) Energy Variance
(c) Cell Energy Ratio
(f) Layer Energy
Figure 10: Histograms of various shower shape variables
10a-
10
SHhv SUNKO
10a
Sunt
SUNA2
10r
SUNvi
Donglt
10
10-7
10-2
M
2.0
10
Figure 11: Histograms of layer centroids
Sunw3
SuNti
Donsty
10-1
10
10-1
10-4
10-1
10-1
10
1
0.0
0.5
1.3
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
2.0
0.5
1.0
15
2.0
0%s
Figure 12: Histograms of layer widths
A.3 Point Cloud Generative Models
PointFlow PointFlow Yanng et al. 2019) is flow based model with PointNet-like encoder and continuous normalizing flow (CNF) decoder. Additionally, the latents (encoder outputs) modeled with another CNF to enable sampling. We adapted the PointFlow code to handle variable number of points with masking and masked batch norm. The encoder consists of 1D convolutions with filter sizes 128, 128, 256 and 512, followed by a three-layer MLP with 256 and 128 hidden dimensions convert the point cloud into its latent representation of size 128. The CNF decoder has four conditional concatsquash layers with a hidden dimension of 128 and the latent CNF has three concatsquash layers with hidden dimension 64. The overall architecture has 0.7.M trainable parameters.
SetVAE SetVAE Kim et al. 2021 is a transformer-based hierarchical VAE set-structured data which learns latent variables multiple scales, capturing coarse-to-fine dependency of the set elements while achieving permutation invariance. We set the number of heads to 4, the dimension of the initial set to 64, the hidden dimension 64, the number of mixtures for the initial set to 4, and
15

---

the number of inducing points in the hierarchical setup to [2, 42,488 8, 16, 32). The overall architecture has 0.5M trainable parameters.
Transflowmer Transflowmer is flow-architecture using Real NVP layers (Dinh et al. 2016]. As the events are point clouds of varying cardinality, the coupling layers of the flow are required to be permutation equívariant and able to process a varying number of inputs. To satisfy these constraints, transfrmerss [Vaswani et al. [017] without positional encoding in the coupling layers. The overall architecture consists of 16 coupling layers, each them is parametrised by transformer layers with dmodel 32. The overall architecture has 2. 2. parameters.
We train all the models with 100K training examples.
A.4 Experiments on SUPA datasets
We train point cloud generative models, PointFlow Yang et al. 2019), SetVAE Kim et al. 2021]. and Transflowmer SUPA datasets. In this section, show histogram plots to compare the generative performance across different shower shape variables. For all these plots, the axes limits are chosen according to the ground truth data and generated samples have probability mass outside the shown range.
A.4.1
SUPAv1
Figs.
13 18 show the histograms of various shower shape variables for SUPAv1 and samples generated with PointFlow, SetVAE, and Transflowmer.
10-1
S9 0::4044
1821
10-1
Skid 3.00
0.102
SV (2:00341
(0.00
Dardtt
10
Derstty
2.0
10-1
10
10-
10-4
-7.5
-5.0
-25
0.0
5.0
7.5
7.5
-5.0
-25
0.0
5.0
7.5
Figure 13: Histograms of point distributions for 17, 6, and
16
PP 96 351
If 21:54)
1.000
10a
SV 3V 831
2.00:
107
SV 14: 24.741
1he
10F
1918-120
Don2ty
10-1
Dertity
Danslty
10
10-2
10-1
10
10-3
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
1.0
3.2
3.4
1.5
1.8
2.0
310
161
Figure 14: Histograms of sample means for different features
10
N026 6.111
Pi 6.6
10F
5.70
Donstt
10-1
10-3
10-2
10
10
10:4
10-3
10-4
10-4
10-4
0.5
1.0
2.0
2,5
0.0
1.0
1.5
2.5
3.0
0.0
0.5
1.5
2.0
3.0
(2)
Figure 15: Histograms of sample variance for different features
16

---

107
551
FF 2254)
101
5.94
107
1h 912
10
9218:120
(8:00220))
Dendky
10-1
Denskt
Denaky
10"2
10-3
1073
10-4
-1.0
-0.5
0.0
0.5
-1.
-0.5
0.0
0.5
1.0
1.0
1.2
1.4
2.0
6//
071
Figure 16: Histograms of energy weighted averages
SUNVVE M1
SUPSe 3.00
10
SUMWd 0:0000
IT ESD
01
675
Donsty
10-1
110
Denoty
10-3-
10
100
10
10:1
10-2
10a1
10th
1.0
1.5
2.5
0.0
1.0
1.5
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Figure 17: Histograms of lateral widths
24.
SPPAAI oo 17
50B 1.0
8.50
0.5
6.75
1.5
150
1.5
200
02
050
175
300
120
3.0
3.75
200
0.1
0.
(a) Point dist, Energy feature
(b) Energy Mean
Energy Variance
P-8 oue
300
1.0
(d) Cell Energy Ratio
(e) Layer Energy
Figure 18: Histograms of various shower shape variables
17

---

A.4.2
SUPAv2
Figs. 19 24
show the histograms of various shower shape variables for SUPAv2 and samples generated with PointFlow, SetVAE, and Transflowmer.
10"1
10
2111 6:211
10
10
10
W1
Dondty
10-1
Densts
10
Dansky
10-1
10-1
10
10-
10:4.
-20
-30
10
20
-20
-10
20
1.0
15
20
Figure 19: Histograms of point distributions for 7), Ø, and
SUFAG 1s 801
10a 0:11
SUTRAZ oF 3.00
10f
SUNM2 6.0
10
10
12: 634
10ah
0.322
Dendtt
10-2
10-2
10
10-1
Figure 20: Histograms of sample means for different features
211::0331
10f
0.14) 6011:00011
S4 15 6.51
10-1
17110:0000
10-4
188
Daraty
Duraity
10-2
10-1
10-1
10
10
10-1
Figure 21: Histograms of sample variance for different features
SUPKu 1s 8.00 1131
SUBNG 1s 0400
2.411
10>
ra Th 0.18)
SUNke cd 3.000
Daraty
10-1
0:20
10--
10
10>2
10-1
10-9
10
10
10
0/
Figure 22: Histograms energy weighted averages
18

---

10c
5.00
2718
10
Dendte
10-1
10--
10-1
10
10-1
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Pye
Pgk
Figure 23: Histograms of lateral widths
10-11
23-1
10-
15.4
20
12
(a) Point dist. for Energy feature
(b) Energy Mean
(c) Energy Variance
Sannd
14-
15*
0.
0.2
3.0
025
00.0
301 075
15.0
12.1
76.
17.
(d) Cell Energy Ratio
(c) Layer Energy
Figure 24: Histograms of various shower shape variables
19

---

A.4.3
SUPAv3
Figs. 25
30 show the histograms of various shower shape variables for SUPAv3 and samples generated with PointFlow, SetVAE, and Transflowmer.
rE: 0.00
1
830
10-3
3.71
10-1
Dondty
10-1
Denstyy
10.4
Danaky
19-7
10-7
10-1
10
10.1
-70
-10
1.0
20
-20
-10
20
1.0
15
20
Figure 25: Histograms of point distributions for 7), Ø, and
SU 0
SUTRu3 DU: 3.00
SURMO 6.0
69:
10-1
1912::0991
10:1
1211:0008
10-2
0.34
pord
10-1
1.02
10-3
10-4
10-1
10
10-1
10-1
Figure 26: Histograms of sample means for different features
10a
0.101
3341
0.000
10
S4 15 1::31
10:3
SV/21 L.231 (0.33)
6.33)
1321
Deraty
Dursity
10
10-1
10-
10-1
10-4
61
Figure 27: Histograms of sample variance for different features
SUnKu is 8000
SUBNO 1s 0400
SUNei cd 3.000
2431
109
2::33
$121
10-1
0.300
Th 22222
Daraty
Dersily
10-1
"
10-2
10-1
10-
10-4
10
0
10
10-3
0/0
Figure 28: Histograms energy weighted averages
20

---

10
PE E20
10-
5,000
Donsts
Denstyy
10a2
10
10-3
10
10
10a4
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.6
1.5
2.0
2.5
3.0
Pye
Pgk
Figure 29: Histograms of lateral widths
Fatt can Wij
31-1
33%
P:
23-
20
D11
(a) Point dist. for Energy feature
(b) Energy Mean
(c) Energy Variance
10
10
30-1
0'2
5.6
025
00.0
475
15.0
12.1
76.
175
301
(d) Cell Energy Ratio
(c) Layer Energy
Figure 30: Histograms of various shower shape variables
21

---

A.4.4
SUPAv4
Figs. 31
show the histograms of various shower shape variables for SUPAv4 and samples generated with PointFlow, SetVAE, and Transflowmer.
36
10-1
6-211
W1
10-1
10-2
Dondoy
10-1
Densty
10-3
Denaky
10-1
10m
10-4
10"4
-20
-30
10
20
-20
-10
20
1.0
15
20
Figure 31: Histograms of point distributions for 7), Ø, and
10
SU 8
107 0211
SUTRe4 OF: 3.000
10
SUNMA 6.00
0:13:00231
0121
1014
10
1211:900 928
10-2
Dosaty
10
10-5
10-4
Figure 32: Histograms of sample means for different features
10t
PF18:0.33
10²
3:
10f
(.00)
54 15 6511
5V 0120:0.01
10-1
1718::0000
10:1
184
10-1
Darato
Duraity
10-1
11
10
10
10-1
Figure 33: Histograms of sample variance for different features
SUnKe is 8000
Subiv Is 0400
10a 14 0:31
1d: 3.000
1:91
10
3121
10-1
Th
berdty
1
10--
10:2
10-1
10-1
10-4
10-
10--
Figure 34: Histograms energy weighted averages
22

---

PE 9333
PE 630
I0f
9.00
10
2718
Dondky
1o
10-1
10-3
10-3
10-1
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Pye
Pgk
Figure 35: Histograms of lateral widths
Foo can Pij 8o
20-2
10.
MF
10-1
23-4
12
011
(a) Point dist. for Energy feature
(b) Energy Mean
(c) Energy Variance
Samna
13-3
139
00
0.2
5.5
0.6
25
(d) Cell Energy Ratio
(c) Layer Energy
Figure 36: Histograms of various shower shape variables
23

---

A.4.5 SUPAv5
We only consider layer O for SUPAv5. Figs. 37 42 show the histograms of various shower shape variables for SUPAv5 and samples generated with PointFlow, SetVAE, and Transflowmer.
BY D40 28 6.10
2.04)
10-3
Donsts
Dendty
Dansky
20*1
10-4
10-4
10*4
-20
-30
10
20
-20
-10
30
20
1.0
15
20
Figure 37: Histograms of point distributions for 7), Ø, and
SUFAS 18 001
SUTRAS OF 3.00
10'
SU%Md 6.00
02:
10
1232:03311
10:1
12: 614
10-2
255
Donatt
10-2
10:2
10
10
10
Figure 38: Histograms of sample means for different features
PFI8:
3821
10
0.00)
S4 15 641
10-1
10"4
3321
1T 0.222)
Daraty
Daraltt
10-1
10--
10-2
10-a
10
10
10-1
Figure 39: Histograms of sample variance for different features
P- 231) S01112 2:111
PF 11 2.R11
10-l
Dendyy
Dunsky
10-3
10-4
10
10
Figure 40: Histograms energy weighted averages

---

10
9.000
2714
10a1
Dendkr
10-1
10-1
10-2
10
10-1
10-3
10
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.6
1.5
2.0
2.5
3.0
Pye
Pgk
Figure 41: Histograms of lateral widths
FatA 2a0 F130 $13
pawa Whar1 571
33"0
23-
10
(a) Point dist. for Energy feature
(b) Energy Mean
(c) Energy Variance
11-
13a1
0.
6.0
20
66
(d) Cell Energy Ratio
(e) Layer Energy
Figure 42: Histograms of various shower shape variables
A.5 Experiments grid representation of data
In this section we will present some studies on generative modeling with the grid representation of data from SUPA. We discuss about how to downsample the point clouds below. For these studies, generated another version of the dataset with SUPA such that is similar to the CALOGAN dataset, i.e., with three layers and downsampled to resolution in the multiples of 3 96, 12 x 12, and 12 6, for layer 0, 1, and 2, respectively.
Downsampling. For comparison, we downsample the point clouds their corresponding image representation (see Figure([ by first defining the region of interest i.e. a rectangular region for each layer and the number of bins/cells/pixels in both the horizontal (or 7)) and vertical (or Φ) directions. Finally, for each cell, we sum the energy of the points falling within it to get the pixel intensity. We can increase the number of cells in order to get higher resolutions. Figure Ib Ic and Id show the downsampled image representations at resolutions of 3x, 2x and Ix respectively for the shower shown in Figure la We choose Ix to be the same resolution used in CaloGAN (Paganini et al. 2018 (i.e. 12 x 12 for Layer 1).
A.5.1
Validity of SUPA as benchmark with grid representation
We show the comparison of performance of generative models trained over data generated with SUPA and Geant4 in $ 5.3 this section, we extend those studies with more analysis and plots. Figure 43 shows the scatter plot of the average ranks of those models. The average rank for model on dataset obtained by first ranking them with respect to each marginal's discrepancy and then averaging over all the marginals.
55 for a detailed explanation on the marginals and Paganini et al. (0018 for the grid representation based marginals) for GEANT4 and
Further, in Figures 44 49 we show a subset the marginals (see $
25

---

Caleflow
CaloGAN
CaloVAE
2,4
5303 rank
2.2
2.0
14
1,00
1.25
1.50
1.75
2,00
2.50
2.75
geant_rank
Figure 43: Scatter Plot for ranks over different models. Ranking of the models are consistent over both, SUPA and GEANT4, showing the validity of SUPA as a benchmark.
304
20
Deptr-enlrttd enargy
Shower OppTt
Showe Septh Webh
Canda
30"
36"
36-
36*
Shower Ceptria
Showe Depth
Figure 44: Histogram for various marginals for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated from different trained models
SUPA and also the showers generated with different models trained them. These marginal plots illustrate the diversity in various distributions present in data from GEANT4, and, more importantly SUPA. Further, the distributions the generated showers from different models behave similarly on both datasets, reinstating the proposition that a better model on SUPA implies a better model on the detailed GEANT4.
19e7 Crmp -t.
1eer OeAii Co
Figure 45: Histogram for Layer Energy for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated with different trained models.
26

---

Leyrii heaiin.. sufe
Figure 46: Histogram for Layer energy fraction for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated with different trained models.
CARIS
Figure 47: Histogram for Layer lateral width for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated with different trained models.
A.5.2 High-resolution experiments
In this section, show the utility of SUPA beyond using it for training low resolution (similar to the resolution used in CaloGAN, which we call Ix), as well as the limitation of the current models.
1x
2x
3x
1x
3.57
6.35
7.20
2x
6.78
3x
8.29
Table Mean discrepancy metric (see $ A.5.1) 1) for CaloFlow model when trained and tested over different resolutions. Columns correspond the training resolution and rows to the test resolution. The results on the diagonal show that CaloFlow's performance degrades when resolution increases, and the top shows that it is not simply due to the sheer dimensionality of the signal since the model does not leverage structure at high resolution to perform better at low resolution.
We train CaloFlow [Krasse and Shih 2021 with SUPA by downsampling the point clouds at the
higher resolutions of 2x and 3x. Table shows the mean discrepancy metric (see $ A.55.I for the models. We observe the trend that training at higher resolutions result in poorer performance (diagonal terms) general. Further, when the generated samples from the trained models are downsampled to 1x, the performance deteriorates as compared to samples generated from models trained directly with data at resolution.
27

---

Figure 48: Histogram for i for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated with different trained models.
SARM
114
Figure 49: Histogram for Layer sparsity for GEANT4 e+ (top) and SUPA (bottom) vs. showers generated with different trained models.
28


Reviews:
[{"review_id": "siBp99PLM9", "reviewer": "Reviewer_hC4c", "summary": "", "questions": "", "limitations": "- it is not clear how to tune parameters of SUPA to match characteristics of statistical performance of generally recognized simulators. Should we use some Bayesian optimization framework? Or manual try-and-error process? or what?\n\n- The paper is called \"A Lightweight Diagnostic Simulator for Machine Learning in Particle Physics\". What have we been able to diagnose in particle physics using the proposed simulator? I can not find any specific information in the main text of the paper\n\n- The authors submit their paper in NeurIPS 2023 Track Datasets and Benchmarks. It is not clear how we can use the ideas from the proposed paper as some benchmark. I think, first we should formulate some (possibly, predictive) problem. Then test a number of methods on the generated data and provide some accuracy indicators. And then we will be able to demonstrate the results in greater details and understand which predictive ML method is the best for the considered specific problem", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- important practical use case with non-standard type of data (particle showers in detectors)\n- a parametric model SUPA approximating statistical properties of complex simulators\n- generated datasets can be used to pre-train ML models for detection of particles with specific properties", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zVRKqOteAS", "reviewer": "Reviewer_XfFU", "summary": "", "questions": "", "limitations": "The authors didn’t discuss the negative social impact of their work. I suggest the authors to add a limitation section to discuss the limitation of dataset generated by SUPA compared with Geant-4.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "SUPA is 1-4 orders of magnitude faster than Geant-4. It provides a flexible tool to quickly generate toy datasets and test the performance of particle generative models.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "qEbY4mvzGH", "reviewer": "Reviewer_fTbf", "summary": "", "questions": "", "limitations": "A minor comment would be to develop more metrics for testing on SUPA datasets in line with the state of the art of the field\n", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Because of the lightweight framework for generating datasets, it is very convenient for testing the adaptability of generative models to different detector configurations.  There is a clear value add to the field for SUPA to enable developing more generalizable generative models and the potential for exploring foundation generative models.  The authors should lean into this more by providing datasets that overlap as much as possible with existing work to be a superset — particularly with the 3 Calo Challenge datasets \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "iObN1MJkKe", "reviewer": "Reviewer_q153", "summary": "", "questions": "", "limitations": "Major:\n1. The paper lacks novelty and compared to the Geant4, their simulator seems incremental contribution. \n2. I am concerned this is more of a statistical computing paper where a simulator is more of a focus of this paper than developing an actual benchmark dataset which would be hard to generate for ML practitioners.\n3. Appendix is missing from the submission.\n4. Benchmark model evaluations are less than extensive.\n5. Data size is small. Why not create a more exhaustive benchmark?\n\nMinor:\n1. Schematic Fig 2 (left) says vertical lines are slices and layers are a superset of the slices but in line 183 you say layers are a subset of slices.\n2. Footnotes 3 and 4 appear before their main text reference.\n3. line 196: is N not the total number of slices?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Paper is clearly written and easy to follow.\n2. The simulator is straightforward and Python implementation is helpful for ML practitioners.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mv9ufSsXDz", "reviewer": "Reviewer_Y4mj", "summary": "", "questions": "", "limitations": "Yes. The limitations have been adequately described, although some further explanation as described above would be useful.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper provides a significant advance over earlier work, as it is now feasible to produce benchmark datasets for various configurations, since the ML code runs so much faster than GEANT4. This has the potential to greatly expand what is feasible to test, as much larger and more varied datasets can be produced. So, the broader research community can benefit a lot from the work. As this is generating particle showers that would be produced in particle colliders, there don't seem to be any negative ethical or social implications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Paper presents a surrogate model for sampling of particle showers in high-energy partilce detectors, providing realistic QoI data to supplement detailed simulators.
It includes physical bias into the surrogate ansatz.
The paper appears to have potentially high impact within its targeted audience
The reviews indicate that the limitations by approach and in generated datasets could have been addressed more comprehensively.
Decision: Accept (Poster)