Learning Particle
Physics by Example
Location-Aware Generative Adversarial Networksfor Physics Synthesis

Yale
Luke de Oliveira, Michela Paganini*, Benjamin Nachman

Jets and their structure

1

HEP Simulation
TheoryHardInteractions (MECalculations)
Hadronization &PartonShowering
Detector Sim. &Interactions
Digitization& Pile-Up…

Outstanding Issues
Full Simulation is slow
Detector simulation can takeO(min/event), and ME
calculations to high order in
perturbation can compete fortotal generation timeTime

Hard to model tail-statistics
To get a few rare events in kinematicallyunfavorable regions, we often have tosimulate a large number of events

Petabytes ofSimulated Data
Large amounts of simulated
data needs to be storedand transferred

DiskSpaceRareEvents

Looking for a Solution
FastPortableSpecializedGenerative Models
Traditional Solutions• Hidden Markov Models• Gaussian Mixture Models• (Restricted) Boltzmann Machines• Helmholtz Machines
More Modern Solutions:• Deep Belief Nets
• Variational Auto-Encoders• Autoregressive Models (PixelCNN, WaveNet, …)• Generative Adversarial Networks

Generative
Adversarial Networks
tries to distinguish real images
from generated images

tries to turn noise into
credible samples

Step 0: Simplified Task

Step 0: Simplified Task

single layer calorimeter

Jets
η=0
η = -∞

η = +∞EnergyΦηΦJet Images
Pixelate

TranslateSingle Jet Image

RotateRe-Grid

Flip

Average of Thousands of Jet Images

References:J. Cogan, M. Kagan, E. Strauss and A. Schwarztman, Jet-Images: Computer Vision Inspired Techniques for Jet Tagging, JHEP 02 (2015) 118 [1407.5675].
L. de Oliveira, M. Kagan, L. Mackey, B. Nachman and A. Schwartzman, Jet-images — deep learning edition, JHEP 07 (2016) 069 [1511.05190].
L. G. Almeida, M. Backovi´c, M. Cliche, S. J. Lee and M. Perelstein, Playing Tag with ANN: Boosted Top Identification with Pattern Recognition, JHEP 07 (2015) 086 [1501.05968].
P. T. Komiske, E. M. Metodiev and M. D. Schwartz, Deep learning in color: towards automated quark/gluon jet discrimination, [1612.01551].
J. Barnard, E. N. Dawe, M. J. Dolan and N. Rajcic, Parton Shower Uncertainties in Jet Substructure Analyses with Deep Neural Networks, [1609.00607].
P. Baldi, K. Bauer, C. Eng, P. Sadowski and D. Whiteson, Jet Substructure Classification in High-Energy Physics with Deep Neural Networks, Phys. Rev. D93 (2016), no. 9 094034 [1603.09349].

Samples• Jet images produced with Pythia 8.219• Two processes: QCD vs boosted W from W’—>WZ• pT range = [250, 300] GeV• Jet clustering with anti-kt (R=1.0) with FastJet 3.2.1• Trimming by re-clustering the constituents into

R=0.3 kt subjets and dropping those with <5% jet pT

Benchmark Models• DCGAN — convolutional layers in both G and D• FCGAN — fully-connected layers in both G and D• HYBRIDGAN — a combination of the two:
real images

Discriminator
DCGAN
FCGANGeneratorz

DCGAN
FCGANavgyyaux

LAGAN(Location Aware Generative
Adversarial Network)

Starting from ACGAN
Request a specific class(here, W or QCD)

Correctly classify samples(AuxiliaryClassifier GAN)

Layer Comparison

Convolutional Layer (weight sharing)Layer Comparison

Convolutional Layer (weight sharing)

Locally-Connected Layer

Architecture Guidelines
recommended in GAN literature•

Activation:to induce sparsity
Leaky ReLU•

Batch Normalization:•Mini-batch Discrimination:
helps reduce
mode collapse

ReLU

helps with highdynamic range

High Entropy

Low Entropy

Keras Generator
def generator(latent_size, return_intermediate=False):# this is the z space commonly referred to in GAN papers
latent = Input(shape=(latent_size, ))# this will be our label
image_class = Input(shape=(1, ), dtype='int32')
emb = Flatten()(Embedding(2, latent_size, input_length=1,init=‘glorot_normal')(image_class))# hadamard product between z-space and a class conditional embedding
h = merge([latent, emb], mode='mul')
loc = Sequential([# DCGAN-style project & reshape,Dense(128 * 7 * 7, input_dim=latent_size),
Reshape((7, 7, 128)),# block 1: (None, 7, 7, 128) => (None, 14, 14, 64),
Conv2D(64, 5, 5, border_mode='same', init='he_uniform'),
LeakyReLU(),
BatchNormalization(),UpSampling2D(size=(2, 2)),# block 2: (None, 14, 14, 64) => (None, 28, 28, 6),
ZeroPadding2D((2, 2)),LocallyConnected2D(6, 5, 5, init='he_uniform'),
LeakyReLU(),BatchNormalization(),UpSampling2D(size=(2, 2)),])# block 3: (None, 28, 28, 6) => (None, 25, 25, 1),
LocallyConnected2D(6, 3, 3, init='he_uniform'),
LeakyReLU(),
LocallyConnected2D(1, 2, 2, bias=False, init='glorot_normal'),Activation('relu')

fake_image = loc(h)return Model(input=[latent, image_class], output=fake_image)Performance Evaluation

Discriminator Output
Output of the discriminator (primary task) at the epoch that waschosen to analyze the LAGAN performance in the paper:•Ideal scenario: D outputs
1/2 for all samples,regardless of their origin•

For a well trained D, thiscorresponds to G
producing realistic enoughsamples, so that D can’ttell them apart from thereal ones

Qualitative:
Pythia vs LAGAN images
Qualitative assessment of generated jet images:
avg
Pythiaavg
LAGAN

The LAGAN is not simply memorizing the training set
5 random Pythiaimages
their nearest neighborin the LAGAN
generated dataset

Physical DistributionsCheck: does the LAGAN recover the true data distribution as
projected onto a set of meaningful 1D manifolds? ✓— signal— backgroundDiscriminator output
versus mass and pT
Primary classifier’s performance is quite stable over whole range of m, pT

Excellent performance for aux task. Small m and pT dependent features

Performance

Distribution of any number ofmeaningful manifolds

Performance comparison among architectures

Distribution
Trueof generated
data
imagesdistributionof jet
imagesclassFast

Portable

SpecializedFastz

Specialized pTm…

Trained LAGAN weights:Portable

20 MBSource code:
480 kBGenerator

Peeking throughthe Generator
Location of physical features to distinguish W bosons from QCDinPNA

ih aytinGALHow early do these features develop within the layers of the generator?
6 output channels in first locally-connected hidden layer

6 output channels in second locally-connected hidden layer

Peeking throughthe Discriminator

Convolutional filters in the first layer of the discriminator

their convolved version with the diﬀerence
between the average signal and backgroundgenerated image

their convolved version with the diﬀerence
between average Pythia and average
generate image

GAN images forclassification•

GAN produces very signal or very background looking images•

Unlikely to explore gray area between labels because it’sunfavorable under loss formulation•

Label flipping should help addressing this issue — maybe notaggressive enough•Output of aux task
clearly shows thisissue•

GAN images still
useful for data
augmentation

Conclusion & Outlook•

Resized problem of generating simulation to first address
the simplified 2D case, using jet images•

GANs — validated as a tool for scientific simulation•

Interdisciplinary collaboration needed‣ out-of-the-box, vanilla solutions are not enough•

LAGAN — optimized for sparse, unbounded, and highly
non-linearly location-dependent data distributions

Reproduce This!
•This paper on the arXiv [1701.05927]•

Github repository: centralized location with all links•

Download the training dataset of Pythia jet images, or generatethem yourself using this Docker image•

Train a DCGAN, fully-connected GAN, hybrid GAN, or LAGAN using
the code we provide in models, or load our pre-trained weights•Use the jupyter notebook in analysis to make plots like ours