Can neural networks emulate physics?
Towards a Cosmology Emulator using
Generative Adversarial Networks

Mustafa Mustafa

Data Analytics Services, NERSC
Berkeley Lab.
AI@SLAC, SLAC, Stanford University
10/03/2017

The inverse problem of cosmology
The sky surveys collected by observatory experiments pose
an inverse problem:
given images of the sky and the “standard model” ofcosmology (ΛCDM), can we extract the cosmologicalparameters of our universe?

Mustafa Mustafa (2017)2

Cosmo Convergence Maps

Weak lensing convergence maps κ(ν) for a ΛCDM cosmological model.
Mustafa Mustafa (2017)3

Generative Models

The central problem of generative models is that given a data
distribution

can one devise a generator

such that the generated distribution?

Mustafa Mustafa (2017)5

Generative Models

The central problem of generative models is that given a data
distribution

can one devise a generator

such that the generated distribution?
Our information about

Mustafa Mustafa (2017)

comes from an independent and identically distributed sample
which is assumed to have the same distribution as.

6Generative Models

blog.openai.com/generative-models

Mustafa Mustafa (2017)7

Density Estimation
Creative Commons Attribution-Share Alike 3.0 Unported license

True Distribution
Random samples
Estimator

Achieving a high fidelity generation scheme amounts to the construction of a density estimator of thetraining data.
Mustafa Mustafa (2017)8

Generative Adversarial Networks
GANs, Goodfellow et al.arXiv:1406.2661

Mustafa Mustafa (2017)9

Generative Adversarial Networks
GANs, Goodfellow et al.arXiv:1406.2661

Update discriminator parameters

Mustafa Mustafa (2017)

Target10

Generative Adversarial Networks
GANs, Goodfellow et al.arXiv:1406.2661Update discriminator parameters

TargetUpdate generator parameters

Mustafa Mustafa (2017)11

Generative Adversarial Networks
GANs, Goodfellow et al.arXiv:1406.2661

Update discriminator parameters

TargetUpdate generator parameters

Mustafa Mustafa (2017)12

Generative Adversarial Networks – Loss function
Minimax game formulation (saturating):

Mustafa Mustafa (2017)13

Generative Adversarial Networks – Loss function
Minimax game formulation (saturating):

Ian Goodfellow arXiv:1701.00160

Mustafa Mustafa (2017)14

Generative Adversarial Networks – Loss function
Minimax game formulation (saturating):

Ian Goodfellow arXiv:1701.00160

Heuristic loss function (non-saturating):

Mustafa Mustafa (2017)15

Deep Convolutional Generative Adversarial Networks (DCGAN)

DCGAN architecture, Radford, Metz and Chintala arXiv:1511.06434

Mustafa Mustafa (2017)16

Deep Convolutional Generative Adversarial Networks (DCGAN)
DCGAN architecture, Radford, Metz and Chintala arXiv:1511.06434

DCGAN generated celebrity face images
Mustafa Mustafa (2017)17

Deep Convolutional Generative Adversarial Networks (DCGAN)
DCGAN architecture, Radford, Metz and Chintala arXiv:1511.06434

Interpolation in the latent space. Rotations are linear!
Mustafa Mustafa (2017)18

Deep Convolutional Generative Adversarial Networks (DCGAN)
DCGAN architecture, Radford, Metz and Chintala arXiv:1511.06434

Arithmetics in the latent space
Mustafa Mustafa (2017)19

Generative Adversarial Networks

Mustafa Mustafa (2017)20

Evaluation of Generative Models
How close isMustafa Mustafa (2017)

to?

21

Evaluation of Generative Models
How close isto?

We think that when it comes to practical applications of generativemodels, such as in the case of emulating scientific data, the criterion
to evaluate generative models is to study their ability to reproduce thecharacteristic statistics which we can measure on the original dataset.

Mustafa Mustafa (2017)22

Convergence Maps First Order Statistics

Kolmogorov-Smirnov two tailed test yields p-value >0.999
Mustafa Mustafa (2017)23

Fourier Spectral Analysis
quora.com/Whats-the-use-of-Fast-Fourier-Transform

Mustafa Mustafa (2017)24

Fourier Spectral Analysis
https://xkcd.com/26/

Mustafa Mustafa (2017)26

Fourier Spectral Analysis: Power Spectrum

Bands areMustafa Mustafa (2017)27

Fourier Spectral Analysis: Power Spectrum

Mustafa Mustafa (2017)28

Fourier Spectral Analysis: Power Spectrum

Kolmogorov-Smirnov# moments

p-value242> 0.995

6> 0.93

Mustafa Mustafa (2017)29

Non-Gaussian Corrections
The power spectrum captures the Gaussian structures in the images. However, gravity produces
non-Gaussian structures

Mustafa Mustafa (2017)30

Non-Gaussian Corrections
The power spectrum captures the Gaussian structures in the images. However, gravity produces
non-Gaussian structures

The three Minkowski Functionals are sensitive to the higher order correlations.
Mustafa Mustafa (2017)31Non-Gaussian Corrections

Mustafa Mustafa (2017)32

Non-Gaussian Corrections

Kolmogorv-Smirnov# thresholds

p-value34>0.999

16>0.97

6>0.6

10.32

Mustafa Mustafa (2017)33

Generative Models for Emulating Scientific Data

Model parameters

Mustafa Mustafa (2017)

Physical Model

Data of interest,
images or otherwise.

34

Generative Models for Emulating Scientific Data
Random seed(s)Model parameters

Mustafa Mustafa (2017)

Physical Model

Data of interest,
images or otherwise.

35

Generative Models for Emulating Scientific Data
Random seed(s)Model parameters

Physical ModelHow close
can it get?
Generative Model
Random vector

Mustafa Mustafa (2017)36

Generative Models for Emulating Scientific Data
Random seed(s)Can we use GANs to build parametric density
estimatorsof scientific data?
Physical Model

Model parameters?
Generative Model
Random vector

Mustafa Mustafa (2017)37

Generative Models for Emulating Scientific Data
Random seed(s)Can we use GANs to build parametric density
estimatorsof scientific data?
Physical Model

Parameter space?
Generative Model
Random vector

Mustafa Mustafa (2017)38

Generative Models for Emulating Scientific Data
Random seed(s)Parameter space

Can we use GANs to build parametric density
estimatorsof scientific data?
Physical Model

Generative Model
Random vector

Such generators would likely exclude regions in parameter space where thephysical model
exhibits critical behavior.
Mustafa Mustafa (2017)39

CaloGAN

GEANT

CaloGAN: Simulating 3D Calorimeter Showers using GANs

Paganini, de Oliveira and Nachman arXiv:1705.02355

Mustafa Mustafa (2017)40

Summary and Outlook➔➔

We have shown with statistical confidence that GANs can emulate ΛCDM cosmologicalmodel convergence maps
➔ Fourier spectrum of generated maps match that of a validation dataset➔ Non-Gaussian structures are discovered and emulated by the generator
Deep generative models have the potential of creating high-fidelity computationallyinexpensive emulators of scientific data.

Mustafa Mustafa (2017)41

Summary and Outlook➔

We have shown with statistical confidence that GANs can emulate ΛCDM cosmologicalmodel convergence maps
➔ Fourier spectrum of generated maps match that of a validation dataset➔ Non-Gaussian structures are discovered and emulated by the generator

Deep generative models have the potential of creating high-fidelity computationallyinexpensive emulators of scientific data.
➔ Manuscript: “Creating Virtual Universes using Generative Adversarial Networks”, Mustafa,Bard, Lukic, Al-Rfou, Bhimji, arXiv:1706.02390➔Mustafa Mustafa (2017)42

Summary and Outlook➔

We have shown with statistical confidence that GANs can emulate ΛCDM cosmologicalmodel convergence maps
➔ Fourier spectrum of generated maps match that of a validation dataset➔ Non-Gaussian structures are discovered and emulated by the generator

Deep generative models have the potential of creating high-fidelity computationallyinexpensive emulators of scientific data.
➔ Manuscript: “Creating Virtual Universes using Generative Adversarial Networks”, Mustafa,Bard, Lukic, Al-Rfou, Bhimji, arXiv:1706.02390➔➔

Future lines of work:➔ investigate the ability of GANs to interpolate in the parameter space of physical models➔ multi dimensional data: 1 & 3D (see CaloGAN), time dependence, “sequential data”➔ using NN interpretation techniques to gain insight in what these networks are learning

Mustafa Mustafa (2017)43