SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Atul Kumar Sinha
University of Geneva atul.sinha@unige.ch
Daniele Paliotta
University of Geneva
daniele.paliotta@unige.ch
Báint Máté
University of Geneva balint.mate@unige.ch
John A. Raine
University of Geneva, CERN johnny.raine@cern.ch
Tobias Golling
University of Geneva, CERN  
tobias.golling@cern.ch
Deep learning methods have gained popularity in high energy physics for fast modeling of particle showers in detectors. Detailed simulation frameworks such as the gold standard GEANT4 are computationally intensive, and current deep generative architectures work on discretized, lower resolution versions of the detailed simulation. The development of models that work at higher spatial resolutions is currently hindered by the complexity of the full simulation data, and by the lack of simpler, more interpretable benchmarks. Our contribution is SUPA, the SUrrogate PArticle propagation simulator, an algorithm and software package for generating data by simulating simplified particle propagation, scattering and shower development in matter. The generation is extremely fast and easy to use compared to GEANT4, but still exhibits the key characteristics and challenges of the detailed simulation. The proposed simulator generates thousands of particle showers per second on a desktop machine, a speed up of up to 6 orders of magnitudes over GEANT4, and stores detailed geometric information about the shower propagation. SUPA provides much greater flexibility for setting initial conditions and defining multiple benchmarks for the development of models. Moreover, interpreting particle showers as point clouds creates a connection to geometric machine learning and provides challenging and fundamentally new datasets for the field.
In order to understand the fundamental building blocks of nature, High Energy Physics (HEP) experiments involve highly energetic particle collisions. These collisions cause particles to decay, and the identification of the resulting decay particles is of key importance to develop and confirm new physics/theories. This is enabled through an electromagnetic or hadronic calorimeter. As particles interact with the calorimeter, they split into multiple other particles, forming a particle shower. The nature of the generated shower depends on the specific material and geometry of the calorimeter, which are carefully selected and driven by physics theory and practical considerations. The generated shower deposits energy in active layers of the calorimeters, and provides energy and location measurements from the particles produced in the cascade.
Precise computer simulation is central to a better understanding of experimental results in HEP. Simulation is especially useful for the task of event reconstruction, where the deposited energy in the calorimeter is used to identify the particle that originated a given shower. The pattern of the deposited energy depends on the specific type of the (intermediate) particles, the initial energy and incidence angle, as well as the specific geometry and shape of the detector material, among other factors. The state-of-the-art for this kind of simulations is GEANT4 [Agostinelli et al., 2003], a Monte Carlo toolkit for modeling the propagation of particles through matter.
While detailed simulation through GEANT4 provides fine-grained shower generation and captures the underlying distributions accurately, this software consists of more than 3.5 million lines of  $\mathrm{C + + }$ , is computationally very expensive, and requires specific domain knowledge to be set up and tuned. This makes it cumbersome and expensive to produce the amount and diversity of data needed to speed up machine learning research on these topics. Table 1 compares GEANT4 with our proposed simulator SUPA.
Our key contributions to ease these issues are:
- We introduce SUPA: a fast, easy to use simulator for simple particle propagation, scattering and shower development in matter (see § 3).
- We structure the simulator in a way that allows to easily change the complexity of the underlying shower development as well as the properties of the showers in terms of multiplicity, energy range, structure (see § 4.1).
- We use SUPA to generate data that highlights the limitations of current machine learning approaches for fast shower simulation (see § 5.2). We experimentally demonstrate that SUPA can be used as a proxy for detailed simulation, while being easy to tune and up to 6 orders of magnitude faster than GEANT4.
1.1 Image Representation
Existing approaches for generating particle showers [Paganini et al., 2018, Krause and Shih, 2021] focus on a quantized representation of the calorimeter hits, binning them into discrete pixels, or cells, and setting the pixel intensity to the sum of the energy deposited. The upside of this approach is that, due to the structure of existing calorimeters, the data takes the form of low resolution images and standard computer vision models can be directly applied. However, as we increase the resolution of the quantized representations in order to preserve more information, the resulting images become exceedingly sparse, leading to difficulties in training existing machine learning architectures. Figure 1b, 1c, and 1d show the downsampled image representations at resolutions of  $3\mathrm{x}$ ,  $2\mathrm{x}$  and  $1\mathrm{x}$  respectively for the shower shown in Figure 1a. We choose  $1\mathrm{x}$  to be the same resolution as used in CaloGAN [Paganini et al., 2018] (i.e.  $12 \times 12$  for Layer 1).
It is also important to notice that, when decreasing the resolution of the data and looking at showers, while the signal to model is of lower dimension and less sparse, the underlying structure and relationships between hits are lost. This requires models to implicitly learn equivariances from multiple examples, while also introducing artificial structure arising from the chosen detector geometry. For example, with a shower incident on the surface of a detector, a slight translation in either direction does not change the properties of the shower. However it still has an impact on the resulting datasets, where for small shifts it is not translationally equivariant. Having access to the underlying energy depositions would therefore greatly reduce the dependence on the geometry of the detectors in the models, and allow generative models to learn the underlying structure from the physical process directly.
1.2 Point Cloud Representation
Many popular point-cloud datasets [Wu et al., 2015, Chang et al., 2015] in the machine learning community contain shape representations of physical objects, such as points sampled uniformly from the surfaces of chairs, tables, etc. These datasets enjoy many nice properties, e.g. locality and redundancy, that calorimetry datasets do not. Moreover, existing generative models for point clouds rely on these simplifying assumptions and therefore their performance on more diverse datasets is still relatively unexplored. In the case of particle showers in calorimeters, dealing directly with the point cloud representation provides an optimal description of the underlying physics processes. However,
(a) Point Cloud
(b)  $3\mathrm{x}$
(c)  $2\mathrm{x}$
Figure 1: An example shower (single layer) shown at different resolutions
(d) 1x
due to the detailed simulation resulting from GEANT4, the number of points per shower represents a huge computational challenge.
A key objective of our synthetic simulator is to bridge this gap by allowing for quicker ground truth data generation for different initial settings, thus providing a way of modulating the complexity of the dataset, leading to more interpretable benchmarks. This would also make it possible to take gradual steps in developing the models and work out which of the many complexities in the GEANT4 datasets are hardest to model.
Shower simulation with GEANT4 requires a complete description of the geometry and material of the detector and simulates all interactions between the initial particle, any subsequently produced particles through decays or emissions, and the detector material. The upside of using GEANT4 for simulation is that it is very accurate and represents the underlying physical phenomena precisely, but it comes at the cost of being computationally very expensive. Moreover, it requires expertise to define the configuration and a multitude of expert hours to set it up for new detector geometries. To reduce the computational resources required to simulate particle physics collisions and their energy depositions in detectors, deep generative models have been applied to generate particle showers. Most of these models [Abhishek et al., 2021, Krause and Shih, 2021, Paganini et al., 2018] employ the dataset first introduced in CaloGAN [Paganini et al., 2018] which is based on a simplified calorimeter inspired by the ATLAS Liquid Argon (LAr) electromagnetic calorimeter.
2.1 Datasets and calorimeter structure
For the CaloGAN dataset, the calorimeter is cubic in shape with each dimension being  $480\mathrm{mm}$  and no material in front of it. The volume is divided into three layers along the radial  $(z)$  direction with varying thicknesses of  $90\mathrm{mm}$ ,  $347\mathrm{mm}$ , and  $43\mathrm{mm}$ , respectively. These layers are further segmented into discrete cells, with different sizes for each layer:  $160\mathrm{mm} \times 5\mathrm{mm}$  (first layer),  $40\mathrm{mm} \times 40\mathrm{mm}$  (second layer) and  $40\mathrm{mm} \times 80\mathrm{mm}$  (third layer). Each layer can be represented as a single-channel two-dimensional image with pixel intensities representing the energy deposited in the region. The final read-out has the resolution of  $3 \times 96$ ,  $12 \times 12$ , and  $12 \times 6$ . For this dataset, three different particle types were considered, namely, positrons, charged pions and photons. Further, the particles were configured to be incident perpendicular to the calorimeter with initial energies uniformly distributed in the range between  $1\mathrm{GeV}$  and  $100\mathrm{GeV}$ .
Several other calorimeter geometries and configurations are employed across the literature. Erdmann et al. [2019] consider a calorimeter configuration motivated by the CMS High Granularity Calorimeter (HGCAL) prototype, while Belayneh et al. [2020] study a calorimeter based on the geometric layout of the proposed Linear Collider Detector (LCD) for the CLIC accelerator. Buhmann et al. [2021b,a] investigate the prototype calorimeter for the International Large Detector (ILD) which is one of the two proposed detector concepts for the International Linear Collider (ILC). The Fast Calorimeter Simulation Challenge 2022 (CaloChallenge2022) Faucci Giannelli et al. [2022c,a,b] was proposed to spur the development and benchmarking of fast and high-fidelity calorimeter shower generation using deep learning methods. The challenge proposed three datasets, ranging in difficulty from easy
<sup>1</sup>The ATLAS calorimeter has e.g. a more complex geometry, with the cells having accordion shaped electrodes to maximise the active volume
to medium to hard. the difficulty is determined by the dimensionality of the calorimeter showers i.e. number of layers and number of voxels in each layer. In a different application setting, Erdmann et al. [2018] model a calorimeter response for cosmic ray-induced air showers in the Earth's atmosphere, producing signals in ground based detector stations. In contrast to the CaloGAN calorimeter, it has a single readout layer with  $9 \times 9$  cells, each cell corresponds to a detector unit placed with a spacing of  $1500m$ . Although many different detector technologies are considered in the various models, the common aspect between the models and the corresponding datasets is the projection of the spatial signal onto discretized cells in 2D planes or a generalized 3D volume. The projection is done either directly while designing the calorimeter geometry, or as a post processing step. Further, the number of cells and thus the resolution of the final read-out is usually kept small. This is done either to simplify the data for training the models, or in order to speed up the simulations.
Due to the different geometries used for generating the datasets, it is not a straightforward task to compare the performance of the various models. As the developments of each shower are completely dependent on the detailed model of the detector in GEANT4, although the same initial particles can be simulated, it is very difficult to draw parallels between two different datasets, as it is not possible simply to change the representation of the data from one model to look like that of another.
2.2 Deep Generative Models
The applications of deep generative modeling to calorimeter simulation have so far almost entirely focused on Generative Adversarial Networks (GANs, Goodfellow et al. 2014) and Normalizing Flows [Rezende and Mohamed, 2016]. CaloGAN [Paganini et al., 2018] was the first application of GANs to a longitudinally segmented calorimeter. The approach is based on LAGAN [de Oliveira et al., 2017], a DCGAN [Radford et al., 2016]-like architecture, that is able to synthesize the shower images. The generator outputs a gray-scale image for each layer in the calorimeter, with each output pixel representing the energy pattern collected at that location. The architecture is also complemented with an auxiliary classifier tasked with reconstructing the initial energy  $E$ .
Krause and Shih [2021] improve on CaloGAN with CaloFlow, a normalizing flow architecture to generate shower images. CaloFlow is able to generate better samples than previous approaches based on GANs and VAEs, while also providing more stable training. Inductive CaloFlow (iCaloFlow) Buckley et al. [2023] extends the CaloFlow model to higher granularity detector geometries. It utilizes a teacher-student distillation to increase sampling speed without loss of expressivity.
Previous work by ATLAS collaboration [2018] proposes to use a Variational Autoencoder on the flattened images [Kingma and Welling, 2014]. Both the encoder and decoder consist of an MLP conditioned on the energy of the initial particle. In addition to the reconstruction and KL-terms, the authors also optimize for the overall energy deposit in both in the individual layers and in the overall system.
Buhmann et al. [2023] proposes a diffusion-based generative model representing one of the initial methods devised for effectively processing the point cloud representation of calorimeter showers.
3 Propagation Model
SUPA generates data that imitates the propagation and splitting/scattering of point particles. To generate an event, a point particle is initialized at the origin of the 3D space with an initial energy  $z$  value and a velocity parallel to the  $z$ -axis. We then define a sequence of 2D planes orthogonal to the  $z$ -axis at  $\{z_0, \dots, z_N\}$  that we call "slices" (or "sub-layers", interchangeably). We assume that the slices are evenly spaced along the  $z$ -axis, with a gap of  $\Delta z$  between consecutive slices. As the initial particle reaches the first predefined  $z$ -slice at  $z_0$ , it has 3 options (see Figure 2): stop and deposit its energy with stopping probability  $p_{\mathrm{stop}}$ , split into two with splitting probability  $p_{\mathrm{split}}$ , or simply continue moving with the same velocity with pass-through probability  $p_{\mathrm{pass}} = 1 - (p_{\mathrm{stop}} + p_{\mathrm{split}})$ . Then:
- If the particle splits, two particles are generated at the splitting location with energy and velocity features that ensure momentum conservation, i.e., that the sum of momentum
${}^{2}$  A continuous attribute obeying simple conservation laws at each splitting,which can be interpreted as an equivalent for the energy of the particles in a shower. As such,we refer to this property as the energy.
Figure 2: Schematic summary of a propagation event generated by SUPA. (left) Splitting events can happen at the black vertical lines, called slices. See § 3 for a detailed explanation of the process. (right) Each time a particle crosses a pre-defined "slice", it either splits in two, stops and deposits energy, or passes through unperturbed.
vectors of the new particles is equal to the decaying one. The original particle is removed from the propagation model and the two new ones continue propagating with their respective velocities. The splitting angle  $\theta$  and deviation angle  $\epsilon$  are sampled from distributions  $p_{\theta}(.)$  and  $p_{\epsilon}(.)$ , respectively.
- If the particle stops, its energy is deposited and the energy value and the location of the hit are recorded.
- If the particle passes through, it simply continues with its original velocity.
See Algorithm 1 for the pseudocode of this process. We can think of the process as a detector layer being positioned at the  $z_0$ -slice recording energy deposits. After the slice at  $z_0$ , the model either contains 0,1 or 2 particles still propagating. As they reach the additional slices at  $\{z_1, \dots, z_N\}$  the above procedure is repeated: whenever a particle is at a  $z$ -slice, it either passes, deposits its energy or decays.
After all particles crossed the last slice at  $z_{N}$  we collect all energy deposits (energy value and 3D location). Instead of building a dataset from the exact 3D location of these deposits, we partition the slices into subsets we call layers. We refer to the partitioning scheme as layer configuration.
We then build an event from the energy deposit  $E$  and the  $(x, y)$ -coordinate of all hits for all slices within a given layer  $L_i$ , effectively projecting the z-axis into a single point. An event then consists of  $N$  2D point clouds, one for each layer, with a single scalar feature per point. Intuitively, the slices  $z_i$  are the positions where the particle can split and stop (granularity of the simulation), and the layers  $L_i$  correspond to the detector layers, where the readout happens. Figure 2 contains a schematic summary for the propagation process.
Having complete control over the algorithm, we can increase/decrease the complexity of the generated data by adjusting the distribution of the properties of the initial particle, the number and location of the slices  $z_{i}$ , the probabilities  $p_{\mathrm{stop}}, p_{\mathrm{pass}}, p_{\mathrm{split}}$  and the distribution of the properties of the children after a splitting event. This is especially useful to debug and understand the behaviour of machine learning models at different levels for complexity.
Table 1: Comparison of GEANT4 and the proposed model (SUPA)
GEANT4	SUPA
codebase	3.6m lines in C++	few hundred lines in Python
target usage	detailed physics simulations	fast data generation
speed	1772 ms/shower³	0.1-100 ms/shower⁴
Algorithm 1 SUPA - Particle Splitting
Parameters:	Stopping probability: pstop	2: θ ~ pθ(., ε ~ pε(., α ~ U(0,1)
Splitting probability: psplit	3: if α < pstop[l] then
Layer configuration: L1 = {z0, ..., zi1}, L2 = {zi1+1, ..., zi2}, ... Ln = {zi_n-1+1, ..., zn}	4: Active ← False
5: else if pstop[l] < α < pstop[l] + psplit then
6: ω ← U(0,2π)
7: p0 ← p + Δz * (sin(θ/2 + ε), cos(θ/2 + ε))
Slice Gap: Δz	8: p1 ← p - Δz * (sin(θ/2 + ε), cos(θ/2 + ε))
Splitting angle: pθ(.)	9: E{0,1} ← E * (θ/2 ± ε)/θ
Deviation angle: pε(.)	10: T.add(p0, l+1, E0), T.add(p1, l+1, E1)
Input:	Tree T, Position p = (Δη, Δφ), Energy E, Layer l	11: else
Output:	a collection of {pi, Ei}^N of generated particles	12: T.add(p, l+1, E)
13: end if
1: (p, E, l) ← Input, N ← p(δ(n = 2))	14: Return T
Notation. We denote a single shower as  $\mathbf{x} = \{\mathbf{x}_{L_1},\mathbf{x}_{L_2},\dots ,\mathbf{x}_{L_N}\}$ , where  $\mathbf{x}_{L_i}$  denotes the  $i^{\mathrm{th}}$  layer and  $N$  is the total number of layers considered. For the point cloud representation,  $\mathbf{x}_{L_i}$  can be further expanded as  $\mathbf{x}_{L_i} = \{\mathbf{x}_1^i,\mathbf{x}_2^i,\ldots ,\mathbf{x}_{N_i}^i\}$ , where  $N_{i}$  denotes the total number of points in layer  $i$  of the shower (can vary across showers). Further, each point is a vector  $\mathbf{x}_j^i = [\eta_j^i,\phi_j^i,E_j^i]$ , where  $\eta$  and  $\phi$  are the coordinates and  $E$  denotes the energy.
4 Using SUPA
SUPA provides various meta-parameters which affect the data generation and various characteristics of the generated data. This allows to generate shower data with varying levels of complexity and thus enables incremental benchmarking. For instance,  $p_{split}$  affects the number of emerging secondary particles and thus implicitly the number of hits (or points), while  $p_{stop}$  controls the position (in depth) of the observed hits.  $\theta$  controls the overall spread of the hits in the lateral plane and  $\alpha$  controls the dynamic range of the observed energy values. The different configurations for these parameters is analogous to different detector geometry/material or perhaps a different particle.
4.1 SUPA Datasets
We propose five standard configurations and the resulting datasets SUPAv1-5 for our analysis and benchmarking, summarized in Table 2. All configurations have the same initial energy of  $65\mathrm{GeV}$ , initial angle of  $\pi /2$  and initial impact position  $(0,0)$ , i.e. particles are incident perpendicular and at the center of the calorimeter. Section A.2 in the appendix gives more details and also shows the interpretation of these variations with respect to different summary variables. SUPAv1 is deterministic
Table 2: Summary of different datasets generated with SUPA
Dataset	θ	α	\(p_{split}, p_{stop}, p_{pass}\)	# Points	Layer Configuration
SUPAv1	\(\frac{\pi}{24}\)	0	see Fig. 5a	128	L0 : [5,10]
SUPAv2	\(\frac{\pi}{24}\)	0	see Fig. 5b	[1,91]	L0 : [7,20]
SUPAv3	\(\mathcal{U}(\frac{\pi}{32}, \frac{\pi}{16})\)	\(\mathcal{U}(\frac{-\theta}{4}, \frac{\theta}{4})\)	see Fig. 5b	[1,91]	L0 : [7,20]
SUPAv4	\(\mathcal{U}(\frac{\pi}{32}, \frac{\pi}{16})\)	\(\mathcal{U}(\frac{-\theta}{4}, \frac{\theta}{4})\)	see Fig. 5b	[1,84]	L0 : [7,12]
SUPAv5	\(\mathcal{U}(\frac{\pi}{32}, \frac{\pi}{16})\)	\(\mathcal{U}(\frac{-\theta}{4}, \frac{\theta}{4})\)	see Fig. 5c	[5,280]	L0 : [7,12], L1 : [13,28]
<sup>3</sup>The exact simulation time depends on the incident particle and its kinematic properties, as well as the detailed composition of the detector geometry in the GEANT4 model. These numbers are taken for the CaloGAN geometry introduced in Paganini et al. [2018].
Reflects the time to generate directly the point cloud representation, which is more granular than the corresponding GEANT4 generation
with respect to splitting and stopping probabilities as well as in  $\theta$  and  $\alpha$ . It has the same number of points across all events and all the hits have the same energy value. SUPAv2-4 have the same configuration for splitting and stopping probabilities (see Sec. A.2.1), while SUPAv2 is deterministic in  $\theta$  and  $\alpha$ . SUPAv3 is naturally more spread out than SUPAv2 (see Fig. 6b vs. Fig. 6c, and also distributions for  $\eta$ ,  $\phi$ ,  $r$  in Fig. 7, and  $\langle \eta \rangle$ ,  $\langle \phi \rangle$ ,  $\langle r \rangle$  in Fig. 8). SUPAv4 is similar to SUPAv3, except for the layer configuration. SUPAv4 only has a subset of the sub-layers by dropping the last 8 sub-layers from SUPAv3. While SUPAv3 captures the total initial energy in Layer 0, SUPAv4 has some energy leakage which is apparent from the distribution of layer energy  $\bar{E}$  in Fig. 10f. Another interesting structure in shower data is the dynamic range of the energy values in a given shower. A high dynamic range of energy makes learning generative models difficult [Krause and Shih, 2021]. With SUPA, we can control the dynamic range via the parameter  $\alpha$ . SUPAv1 and SUPAv2 have fixed  $\alpha (= 0)$ , thus all the splits are symmetric leading to a low dynamic range (see Fig. 10d for histogram of variance in energy values). The other factor which affects the dynamic range is the layer configuration, more number of slices in a layer would lead to more dynamic range. SUPAv5 has multiple layers and has a higher  $p_{split}$  in the initial slices, thus it is more complex in terms of multiplicity or total number of hits. Please refer to the appendix Sec. A.2 for more details and other canonical SUPA datasets.
4.2 Extensions