Turbo-Sim: a generalised generative model with aphysical latent space

Guillaume Quétantab∗
guillaume.quetant@unige.ch
Vitaliy Kinakha
vitaliy.kinakh@unige.chaMariia Drozdovaabmariia.drozdova@unige.ch

Tobias Gollingb
tobias.golling@unige.ch

Slava Voloshynovkiya∗
svolos@unige.ch

Department of Computer Science, University of Geneva, 1227 Carougeb
Department of Particle Physics, University of Geneva, 1205 Genève

Abstract
We present Turbo-Sim, a generalised autoencoder framework derived from principles of information theory that can be used as a generative model. By maximising
the mutual information between the input and the output of both the encoder andthe decoder, we are able to rediscover the loss terms usually found in adversarial autoencoders and generative adversarial networks, as well as various moresophisticated related models. Our generalised framework makes these models
mathematically interpretable and allows for a diversity of new ones by setting theweight of each loss term separately. The framework is also independent of the
intrinsic architecture of the encoder and the decoder thus leaving a wide choice forthe building blocks of the whole network. We apply Turbo-Sim to a collider physics
generation problem: the transformation of the properties of several particles froma theory space, right after the collision, to an observation space, right after thedetection in an experiment.

1

Introduction

Deep learning models have proven to be promising in tasks consisting in the generation of new data
following a desired, sometimes abstract, theoretical model or in the inference of certain parametersof this model given the observed data. Two predominant frameworks in such tasks are adversarial
autoencoder (AAE) [1] and generative adversarial network (GAN) [2], which have been extensively
studied, modified and redesigned in order to fit the problems they are applied to. In this paper we show
a new framework, Turbo-Sim, that generalises the aforementioned models and gives a mathematicalinterpretation to the usual training strategies.
In both AAE and GAN there are two main variable spaces: a data space and a latent space. The
key underlying idea of Turbo-Sim is to maximise the mutual information between the two spaces,
assuming they are sampled from a joint, usually intractable, probability density. Once the density
is carefully parametrized by deep networks, the mutual information can be decomposed and lower
bounded in a number of ways, the maximisation of which gives rise to several loss terms, including
those of AAEs and GANs, leaving the choice of the exact architecture of the building blocks open.2 It∗
2

G. Quétant and S. Voloshynovskiy are corresponding authors.
A different approach discussed in [3, 4] gives rise to variational autoencoder (VAE) and links it to GAN.

Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021).

also highlights the possibility of training the model in two directions called direct and reverse, whichhelps with the pseudo-invertibility of autoencoder-like models. This is expanded in Section 2.
Generative modelling is a crucial step in the process of scientific analysis, and in particular in particle
physics. Indeed, the huge amount of data collected from experiments such as the Large Hadron
Collider (LHC) needs to be compared with simulated data in order to make statistically meaningfulconclusions. The fast growth of data taken by experiments is accompanied by a fast increase of
the demand for the generation of simulated data, which is generally the bottleneck of the wholeanalysis pipeline. In this regard, machine learning aims to be an alternative to classical Monte Carlogenerative models. Deep networks might for example be used in replacement of low-level feature
simulations such as calorimeter hits [5, 6] or energy deposits in detector cells [7, 8], or of high-levelfeature simulations such as reconstructed observables from raw detector data [9]. In Section 3 we
focus on the latter and make a direct comparison of our framework with the OTUS method [9].3

2

Turbo-Sim formalism

2.1

Base principle

Figure 1: The direct (left) and reverse (right) directions of the Turbo-Sim framework.
Given a dataset composed of pairs {xi , zi }N
i=1 sampled from the joint probability density px,z (x, z)
representing N realisations of the two random variables X and Z, we show that one can maximisea parametrization of the true mutual information I(X; Z) between them in order to get the best
approximation. The true mutual information is often intractable since the true joint density px,z (x, z)
and marginal densities px (x) and pz (z) are usually unknown. In order to approximate the mutual
information, the joint density is first rewritten as:
px,z (x, z) = pz|x (z|x)px (x) = px|z (x|z)pz (z).

(1)

Both conditional densities pz|x (z|x) and px|z (x|z) are then approximated by parametrized ones,qφ (z|x) and pθ (x|z) respectively. The marginal densities are denoted as:q̃φ (z) = Epx (x) [qφ (z|x)],p̂θ (x) = Eq̃φ (z) [pθ (x|z)],p̃θ (x) = Epz (z) [pθ (x|z)],q̂φ (z) = Ep̃θ (x) [qφ (z|x)].

This leads to multiple ways of decomposing the approximated mutual information.
2.2

Approximation for the direct direction

After some algebraic manipulations and by approximating pz|x (z|x) ≈ qφ (z|x), one obtains the two
following lower bounds to I(X; Z) (details can be found in Appendix A and B, as well as in [3, 4]):3

Code is available at https://github.com/quetant/turbo-sim-supplementary.

2hiq (z|x) q̃ (z)Iφz (X; Z) = Epx,z (x,z) log φpz (z) q̃φφ (z) ≥ Epx (x) Eqφ (z|x) [log qφ (z|x)] − DKL (pz (z)kq̃φ (z)), (2)
{z
}
{z
} |
|
Dzz̃ (z̃)−Lz̃ (z,z̃)hip̂θ (x)x
Iφ,θ(Z; X) = Epx,z (x,z) log ppθx(x|z)
KL (px (x)kp̂θ (x)). (3)(x) p̂θ (x) ≥ Epx (x) Eqφ (z|x) [log pθ (x|z)] − D
{z
}
|
{z
} |
Dxx̂ (x̂)−Lx̂ (x,x̂)x
One has thus to train the network in such a way as to maximise both Iφz (X; Z) and Iφ,θ(Z; X) terms,
with a possible weight factor α between them, in order to find the best parameters φ and θ of theencoder and the decoder respectively. This is achieved in the direct direction by minimising thefollowing loss, leading to the left network shown in Fig. 1:

L̄Direct (φ, θ) = Lz̃ (z, z̃) + Dzz̃ (z̃) + α Lx̂ (x, x̂) + α Dxx̂ (x̂).
2.3

(4)

Approximation for the reverse direction

An analogous procedure with the approximation px|z (x|z) ≈ pθ (x|z) gives the following two otherlower bounds to I(X; Z):Iθx (Z; X) ≥ Epz (z) Epθ (x|z) [log pθ (x|z)] − DKL (px (x)kp̃θ (x)),
{z
}
{z
} |
|

Dxx̃ (x̃)−Lx̃ (x,x̃)z
Iφ,θ (X; Z) ≥ Epz (z) Epθ (x|z) [log qφ (z|x)] − DKL (pz (z)kq̂φ (z)),|{z−Lẑ (z,ẑ)

}

|

{z

Dzẑ (ẑ)

(5)
(6)

}

and the reverse direction loss, weighted by β, leading to the right network shown in Fig. 1:
L̄Reverse (φ, θ) = Lx̃ (x, x̃) + Dxx̃ (x̃) + β Lẑ (z, ẑ) + β Dzẑ (ẑ).
2.4

(7)

Generic implementation

The general formalism of the Turbo-Sim framework allows to explain several state-of-the-art autoencoder and generative models. The first crucial point is that the intrinsic encoder and decoderarchitectures are not specified. Indeed they could be any deterministic or stochastic mappers, qφ (z|x)
and pθ (x|z) respectively. Dense network might be used for vector data, convolutional network forimages, graph network for point cloud data, or any more sophisticated architecture for a given specificdata structure. The second point is that by turning on or off certain weights (α, β, γ, ...) of the generalloss:
L̄Turbo (φ, θ) = L̄Direct (φ, θ) + γ L̄Reverse (φ, θ),(8)

one can recover a wide range of common training strategies. Here we allow a factor γ between the twolosses to weight them independently. As a simple example, standard AAE [1] is described by Dz,z̃ (z̃)
approximated by a discriminator network on the latent space and Lx̂ (x, x̂) given by the supervised`p -norm on the data space assuming an exponential distribution for pθ (x|z) ∝ exp −λkx − x̂kpp ,where x̂ is a function of z. Notice that thanks to the α, β and γ weights and because of the λ factors
found in exponential distributions, all eight terms of L̄Turbo (φ, θ) receive an independent factor andthus can be weighted independently.

3

A generative model for particle physics

We apply our generalised Turbo-Sim framework to a collider physics transformation problem. In
particular, we will focus on double top quarks production in proton–proton collision subsequently
3

Figure 2: Distributions of a selection of observables for truth (blue), sampled (orange) and reconstructed (green) data.

decaying into b-quarks and W-bosons which in turn decay into a pair of light leptons and a pairof light quarks, pp → tt̄ → e− ν̄e bb̄ ud¯ as described in [9].4 The final state of this process is thus
made of six particles whose four-momenta are taken to form the Z space. After parton shower,
hadronisation and potential formation of jets in the detectors, the four-momenta assigned to thereconstructed objects are taken to form the X space. In this space, one loses information about theoriginal particles, due to mainly two reasons. Firstly, the neutrino cannot be detected, hence it isindirectly defined by the missing transverse momentum, which must sum up to zero. Secondly, thefour jets cannot be assigned to the four initial quarks since they all decay in statistically identicalshowers in the detector. They are thus treated as independent objects.
Transforming the random distributions of observables well defined by the rules of quantum field
theory into somewhat modified distributions of the same observables due to the stochastic nature of
the experiments is a common problem in particle physics generative modelling. The key point in this
scenario is the word transformation. Indeed, standard simulation tools take the Z space as input andtransform it by applying physically motivated stochastic processes in order to output the X space[10, 11, 12]. On the other hand, usual deep learning models are trained to mimic the same outputs
by taking Gaussian, or any tractable distributions, as input. The paradigm shift introduced in [9] isto rather take this Z space as latent space for the deep network, and thus as input for the generativemodel. We follow the same line to train the full Turbo-Sim network shown in Fig. 1 and select thebest model after a random hyperparameters search.5 Details can be found in the code.
The distributions of a selection of observables resulting from our trained Turbo-Sim model are shownin Fig. 2, where sampled (z̃ and x̃) and reconstructed (ẑ and x̂) data are compared to the truth (z andx). A quantitative comparison of samples with the OTUS method is shown in Tab. 1 by means of theKolmogorov-Smirnov test, which defines a distance between two cumulative distributions. pby , pbz ,E b are the momenta and energy of the b-quark, pjet1, pjet1, E jet1 are the momenta and energy ofyz
the leading jet. These variables, among others, are used during the training. From them we computethe masses of several underlying physical objects in order to verify how well the underlying physicallep
correlations are learnt. mtt is the top quark pair mass, mhad
W is the hadronic W-boson mass, mt ,had
mt are the leptonic and hadronic top quark masses. The reconstructed W-boson and top quark
masses are computed from the four-momentum of the four jets, the electron and the reconstructed
neutrino in the X space. All proper combinations of objects are compared to the true masses thanks
to a chi-square test, the best one being kept. The neutrino longitudinal momentum is computed by−solving m2W = (pe + pν )2 .
We observe that, although the OTUS method looks great at sampling the Z space, the Turbo-Sim
method is better at sampling the X space, which is the main goal of a generative model. In addition,
Turbo-Sim outperforms OTUS when it comes to the reconstruction of decayed objects, implying thatthe underlying physics has been better learnt.
4

The dataset was shared with us and is now publicly available at https://github.com/yiboyang/otus.
The random hyperparameters search was performed on an internal cluster with either nVidia Titan X, P100,
RTX 2080 Ti, RTX 3090 or A100 GPUs, for a maximum of 12 hours.
5

4

Table 1: Kolmogorov-Smirnov distance to truth [×10−2 ] for several observables including those of
Fig. 2. A lower value means a higher accuracy.
Z space

4

X space

Reconstructed physics
mtt

mhad
W

mlep
t

mhad
t

4.43

2.97

7.72

5.20

8.52

5.75

15.8

11.7

14.1

24.9

Model

pby

pbz

E

b

pjet1
y

pjet1
z

E

Turbo-Sim

5.28

7.28

3.96

2.89

10.3

OTUS

1.59

1.23

2.76

3.78

2.39

jet1

Discussion

In this work we presented a new autoencoder interpretation based on the maximisation of the mutual
information between the latent space and the data space. We used this formalism to build a generativemodel for particle physics called Turbo-Sim and showed that it is able to compete with state-of-the-art
methods such as OTUS, even outperforming it in critical tasks. It should be emphasised that theseresults are reached with very basic network building blocks: both the encoder and the decoder are
only made of fully connected layers, as well as the Wasserstein gradient penalty discriminators.
Future implementations with more expressive networks are expected to enhance the performance.
One key improvement would be in the way stochasticity is handled. Instead of simply adding somenoise to the input of the networks, one might use a layer-wise noise implementation following the
EigenGAN architecture [13]. This way, one allows the model to learn how to apply different noise
inputs to different hidden features of the data, thus controlling them separately. Moreover, in contrast
with the OTUS method, we did not include any physically motivated constraints such as momentumalignment or mass conservation alongside our Turbo-Sim loss terms, while yielding comparable
performance.
Another interesting application of such models is to use them for unfolding tasks. Indeed, an important
aspect of particle physics analysis is also to get back from the observed data to the actual physics. In
our language, this means going from the X space to the Z space. Thanks to the paradigm of using aphysically meaningful latent space, i.e. the theoretical distributions of energy and momenta, it turns
out that our Turbo-Sim model is also trained to achieve this task, but its performance is still belowthat of OTUS. However, because of the symmetrical nature of the network that manifests itself withthe direct and reverse direction of training, it should be able to also learn this transformation properly.
Nevertheless, a limitation could come from the data itself. Due to the highly stochastic nature ofdetector observations, the X space shows more smoothed out features than the Z space, which makes
the inference direction harder. Future EigenGAN-like implementations might help handling thisstochasticity, hence we are confident that a more expressive model will be able to perform equallywell in both generation and unfolding tasks.

Acknowledgments and Disclosure of Funding
We would like to thank Sebastian Pina-Otey and Johnny Raine for fruitful discussions, as well as fortheir enlightening comments and feedback. This research was partially funded by the SNF Sinergia
project (CRSII5_193716): Robust Deep Density Models for High-Energy Particle Physics and Solar
Flare Analysis (RODEM).

References[1][2]

Alireza Makhzani et al. Adversarial autoencoders. 2015. arXiv: 1511.05644 [cs.LG].
Ian J. Goodfellow et al. Generative Adversarial Networks. 2014. arXiv: 1406 . 2661
[stat.ML].
[3] Slava Voloshynovskiy et al. Information bottleneck through variational glasses. 2019. arXiv:
1912.00830 [cs.CV].
[4] Slava Voloshynovskiy et al. “Variational Information Bottleneck for Semi-Supervised Classification”. In: Entropy 22.9 (2020). DOI: 10.3390/e22090943.

5

[5]

[6]
[7]
[8]

[9]
[10]

[11]

[12]

[13]

ATLAS Collaboration. Fast simulation of the ATLAS calorimeter system with Generative
Adversarial Networks. Tech. rep. Geneva: CERN, 2020. URL: https://cds.cern.ch/record/2746032.
ATLAS Collaboration. AtlFast3: the next generation of fast simulation in ATLAS. 2021. arXiv:
2109.02551 [hep-ex].
ATLAS Collaboration. Deep generative models for fast shower simulation in ATLAS. Tech. rep.
Geneva: CERN, 2018. URL: http://cds.cern.ch/record/2630433.
Erik Buhmann et al. “Getting High: High Fidelity Simulation of High Granularity Calorimeterswith High Speed”. In: Computing and Software for Big Science 5.13 (2021). DOI: 10.1007/
s41781-021-00056-0.
Jessica N. Howard et al. Foundations of a Fast, Data-Driven, Machine-Learned Simulator.
2021. arXiv: 2101.08944 [hep-ph].
J. Alwall et al. “The automated computation of tree-level and next-to-leading order differential
cross sections, and their matching to parton shower simulations”. In: Journal of High Energy
Physics 2014.7 (2014). DOI: 10.1007/jhep07(2014)079.
S. Agostinelli et al. “Geant4—a simulation toolkit”. In: Nuclear Instruments and Methods inPhysics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment
506.3 (2003), pp. 250–303. DOI: https://doi.org/10.1016/S0168-9002(03)01368-8.
J. de Favereau et al. “DELPHES 3: a modular framework for fast simulation of a genericcollider experiment”. In: Journal of High Energy Physics 2014.2 (2014). DOI: 10.1007/
jhep02(2014)057.
Zhenliang He, Meina Kan, and Shiguang Shan. EigenGAN: Layer-Wise Eigen-Learning for
GANs. 2021. arXiv: 2104.12476 [cs.CV].

6

A

Lower bound to the true mutual information

In this section, we prove that the parametrization of the true mutual information between the X
and the Z spaces I(X; Z) ' Iφz (X; Z) is not only an approximation, but also a lower bound i.e.
I(X; Z) ≥ Iφz (X; Z):px,z (x, z)I(X; Z) = Epx,z (x,z) log
px (x)pz (z)
pz|x (z|x)= Epx,z (x,z) log
pz (z)

pz|x (z|x) qφ (z|x)= Epx,z (x,z) log
pz (z) qφ (z|x)

pz|x (z|x)
qφ (z|x)= Epx,z (x,z) log+ Epx,z (x,z) log
pz (z)qφ (z|x)qφ (z|x)= Epx,z (x,z) log+ Epx (x) DKL (pz|x (z|x)kqφ (z|x))pz (z)qφ (z|x)≥ Epx,z (x,z) log
pz (z)qφ (z|x) q̃φ (z)= Epx,z (x,z) log= Iφz (X; Z),pz (z) q̃φ (z)(9)

where the inequality holds because DKL (pz|x (z|x)kqφ (z|x)) ≥ 0. Maximising the approximatedmutual information ensures that the parametrized density qφ (z|x) is as close as possible to the truexz
density pz|x (z|x). The proves for Iφ,θ(Z; X), Iθx (Z; X) and Iφ,θ(X; Z) follow the exact same logic.

B

Lower bounds to the approximate mutual information

In this section, we derive a lower bound to the parametrized approximation of the mutual information
Iφz (X; Z) by decomposing it for the direct direction as:
Iφz (X; Z) = Epx,z (x,z)

qφ (z|x) q̃φ (z)log
pz (z) q̃φ (z)pz (z)= Epx,z (x,z) [log qφ (z|x)] − Epx,z (x,z) log− Epx,z (x,z) [log q̃φ (z)]q̃φ (z)pz (z)= Epx (x) Epz|x (z|x) [log qφ (z|x)] − Epz (z) log− Epz (z) [log q̃φ (z)]q̃φ (z)= Epx (x) Epz|x (z|x) [log qφ (z|x)] − DKL (pz (z)kq̃φ (z)) + H(pz (z); q̃φ (z))(10)

≥ Epx (x) Epz|x (z|x) [log qφ (z|x)] − DKL (pz (z)kq̃φ (z))' Epx (x) Eqφ (z|x) [log qφ (z|x)] − DKL (pz (z)kq̃φ (z)),
where the inequality holds because H(pz (z); q̃φ (z)) ≥ 0. Analogous derivations can be performed
for the three remaining approximations:pθ (x|z) p̂θ (x)xIφ,θ(Z; X) = Epx,z (x,z) log
px (x) p̂θ (x)pθ (x|z) p̃θ (x)Iθx (Z; X) = Epx,z (x,z) log
px (x) p̃θ (x)qφ (z|x) q̂φ (z)zIφ,θ(X; Z) = Epx,z (x,z) log
pz (z) q̂φ (z)7Societal impacts
Since Turbo-Sim aims to improve the performance of generative models such as AAEs and GANs,
it may lead to any negative usage of these models. However, the framework is here developed forparticle physics simulations, which do not pose any ethical problem. In addition, the goal of such fast
simulator is to minimise the computing resources needed to generated many samples, which is a step
forward in environmental protection, as long as the training process does not compensate for all thegain.

Paper checklist
1. For authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [YES](b) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [YES](c) Did you discuss any potential negative societal impacts of your work? [YES](d) Did you describe the limitations of your work? [YES] (Self-explained by definition of
the scope of the framework and in Sec. 4)
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [YES] (In Sec. 2)(b) Did you include complete proofs of all theoretical results? [YES] (In Sec. 2)
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [YES] (URL in
Sec. 1)(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [YES] (In the provided code)(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [YES](d) Did you include the amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)? [YES] (In Sec. 3)
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [YES] (In Sec. 3)(b) Did you mention the license of the assets? [N/A](c) Did you include any new assets either in the supplemental material or as a URL? [N/A](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [YES] (In Sec. 3)(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A] (No such possibility in particle physics data)
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]8