Deep Learning and Computer Vision
in High Energy Physics
Michael KaganSLAC
SLAC AI Seminar
December 6, 2016

Aspects of Machine Learning (ML) in HEP2	  • Optimization– Bottom line is performance– But can we build new better (simple?) features?
Pythia 8, W'→ WZ,

s = 13 TeV

240 < p /GeV < 260 GeV, 65 < mass/GeV < 95

103

Pixel p [GeV]– Guide and boost performance of ML algorithms
using physics knowledge (i.e. domain specificknowledge)– We don’t want ML to relearn special relativity

102

1

T

[Translated] Azimuthal Angle (φ)• Teaching the learning

T

10
1
0.5

10-1
10-2
10-3

0

10-4
10-5

-0.5

10-6
10-7

-1

10-8
-1

-0.5

0

0.5

10-9

1

[Translated] Pseudorapidity (η)

s = 13 TeV

T

0.6
103

2.0

0.4

102

1

10

1.5

0.2

1 0.0

0.5

10-1

1.0

10-2

0

10-3

0.5

-4

10

10-5

-0.5

0.2
0.4
0.6
0.8

10-6

0.0
0.0
-1

T

Pythia 8, QCD dijets,

240 < p /GeV < 260 GeV, 65 < mass/GeV < 95

Q1
[Translated] Azimuthal
Angle (φ)– Can we extract information about what the ML
is learning?
– Can we use this information to design newvariables?
– Often visualization is a key component

2.5

Pixel p [GeV]• Learning from Learning …(if we can)

0.5

1.0

1.5

Q2

2.0

10-7
2.5 Cell

1.0

Coefficient-810High Energy Physics
What we know3	
  High Energy Physics4	
  What we know

What we know we don’t know(a small sampling)•

How do particles get mass?
– Higgs boson is a piece of the puzzle– But we just found it… and we don’t yet
fully understand it!

•

Why are neutrinos so light?

•

What is Dark Matter?

•

Why is gravity so weak? How does it
work with Quantum Mechanics?
– Extra dimensions of space-time?
– String theory? Something else?

•

Why is there anything at all?
– Existentialism aside…
– Why is the universe filled with matter, anddidn’t annihilate with anti-matter?

•…

The History of the Universe, One Particle at a Time

Working our way backwards

5	
  The Large Hadron Collider at CERN
• proton-proton collisions at anenergy of 8 TeV (soon to be 13 TeV)
• 100 billion protons per bunch

Swiss Alps

• ~3000 bunches of protons• 40 million collisions per second
We work hereE	  =	  mc2	
  

The ATLAS Experiment7	
  ~108 detector channels

Data:~300 MB / sec~3000 TB / year

Weight:7000 tonsSize:
46 m long,
25 m high,
25 m wide

The Intensity Frontier: Exploring the mysteries of the Neutrino

NOvA	
  Experiment	
  

Deep	
  Underground	
  Neutrino	
  Experiment	
  

9	
  

From Theory to Experiment• Standard Model built from
Quantum Field Theory10	
  From Theory to Experiment• Standard Model built from
Quantum Field Theory…	  • Particles interact withcustom detectors– At the LHC, detectors canhave 108 sensors11	
  From Theory to Experiment• Standard Model built from
Quantum Field Theory
…	
  
muon	
  

BoBom	  	
  Quark	  BoBom	  	
  Quark	
  • Particles interact withcustom detectors– At the LHC, detectors canhave 108 sensors• Develop algorithms toidentify particles fromdetector signals
electron	
  

12	
  

From Theory to Experiment• Standard Model built from
Quantum Field Theory…	  • Particles interact withcustom detectors– At the LHC, detectors canhave 108 sensors• Develop algorithms toidentify particles fromdetector signals• Develop small set (~20-40)
engineered features that
describe the interesting
properties of the event…

13	
  Data Analysis in HEP14	
  • Data from 108 sensors→ Reconstruct particles→ Choose feature to extract measurement or search for new particles→ Compare background predictions with potential new signal• Choosing a good feature is tailored to each analysis and paramount tosuccess of the analysis

How Does Machine Learning Fit In?
• In analysis:– Classifying signal from background, especiallyin complex final states– Reconstructing heavy particles and improving
the energy / mass resolution• In reconstruction:– Improving detector level inputs toreconstruction– Particle identification tasks– Energy / direction calibration• In the trigger:– Quickly identifying complex final states• In computing:– Estimating dataset popularity, and determining
how number and location of dataset replicas

15	
  Neural Networks and Deep Learning16	
  Neural Networks17	
  y = (U z + c)

x = input vector

σ(x)	
  =	
  sigmoid	
  funcKon	
  
	
  is	
  the	
  Ac#va#on	
  Func#on	
  	
  z = (W x + b)

• “Typical” neural network circa 2005
• Typical questions of optimization– Which variables to choose as inputs? How correlated are they?
– How many nodes in the hidden layer?

Deep Neural Networks• As data complexity grows, need exponentially large number ofneurons in a single-hidden-layer network to capture all thestructure in the data
• Deep neural networks have many hidden layers– Factorize the learning of structure in the data across many layers• Difficult to train, only recently has this become possible…

18	
  Why did it take so long to train DNN’s?
• Big Data: Large datasets vital fortraining (hundreds of) millions ofparameters• GPU’s: Dramatically increased
speed of training• Improved optimization
algorithms• New regularization techniques:dropout, batch normalization, etc.
• New activation functions, like
Rectified Linear Units
• …

19	
  

Deep NNs in HEP analysis

Nature	  CommunicaKons	
  5,	
  4308	  (2014)	
  Phys.	
  Rev.	
  LeB.	
  114,	
  111801	
  (2015)	
  • Compare dense Deep NN against
BDT’s and shallow NN’s

H→ττ
Signal• Deep NN found to outperform
shallow NN and BDT’s

Background– small but statistically significant gain
over simpler ML algorithms• Physicists are good at doing
physics!
– Typical “high-level” physics variables
are high performing– But Deep NN’s can learn well fromonly low-level inputs20	
  What is deep learning doing?

21	
  • Hierarchical learning of representations• Use low level inputs in smart ways
– e.g. Feed in image pixels, rather than pre-computed features– Learn the structure in the data, rather than engineer it– No explicit need for feature engineering… unless you want to• What deep learning is NOT:
– A silver bullet– Replacement for thinking + domain knowledge– Always better than BDT, SVM, …
– Just feedforward neural networks!
Luke	
  de	
  Oliveira	  Higher Level Representations22	
  • Successive layers build upon information learnedin lower layers to construct progressively higher
level representations of data
OpKmal	  sKmulus	
  
of	
  a	
  given	
  neuron	  hBp://arxiv.org/abs/1112.6209	
  	
  	
  

The Tip of the Iceberg23	
  State of the art inimage recognition andcomputer vision tasks

Luke	
  de	
  Oliveira	  Convolution in 2D24	  Convolutions in 2D

Input image25	
  Convolved image• Scan the filters over the 2D image, producing theconvolved images

What do filters do?

26	
  VGGNet (2014)27	
  • Runner up, 2014 ILSVRC
image recognition challenge– 140M parameters– 2-3 week training time on
4-GPU system

Representation Learning28	
  Layer	  1	  L.	
  Monier,	
  G.	
  Renard,	
  hBps://github.com/holbertonschool/deep-­‐learning	  	
  Representation Learning

29	
  

L.	
  Monier,	
  G.	
  Renard,	
  hBps://github.com/holbertonschool/deep-­‐learning	  	
  Representation Learning

30	
  

L.	
  Monier,	
  G.	
  Renard,	
  hBps://github.com/holbertonschool/deep-­‐learning	  	  Deep Learning for Image Recognition• Deep Convolutional Networks now have super-humanperformance in image recognition (ILSVRC Challenge)31	  Deep Learning and High Energy Physics

32	
  Deep Learning and High Energy Physics• How can we make use of high-performance deeplearning algorithms in HEP?
• Can deep learning find interesting and usefulhigh-level representations of physics data?
– Can they teach us something new?

• Think about our low-level data in news ways thatare amenable to deep learning– Can we frame HEP questions as if they were image
recognition tasks?

33	  Neutrino Identification at NOνA

• Two 2D projections of the interactions• Goal: discriminate between different
neutrino interactions / backgrounds

arXiv:1604.01444	
  

34	
  

Neutrino Identification at NOνA

• Treat 2D projections as images– Convolutional Neural network for imaging
tasks• Make use of GoogLeNet– Use first layers with useful representations forstructures in NOνA detector (e.g. edges, …)– Train with two image inputs, one for each view

arXiv:1604.01444	
  

35	
  

Neutrino Identification at NOνA

Image	  Y-­‐view	  First	
  CONV	
  layer	
  ﬁlters	  arXiv:1604.01444	
  

36	
  

Output	  of	
  convoluKon	  • Convolution filters and outputs show interesting features about how
the NN is providing discrimination• Major gains over current algorithms in νe-CC discrimination:
35% → 49% signal efficiency for the same background rejection

Jets at the LHC37	  Boosted W Jet, R = 0.6

Machine Learning and Jet Physics
2.2

38	
  

2

• Jets are key observables in any event at the LHC
φ

1.8
1.6

– Identify quarks and gluons emitted in high-energy interactions– Identify unstable massive particles: top quark, W, Z, Higgs bosons
1.4
1.2
1

−0.2

0

0.2

(a)
5.8

Boson	
  jet	
  

2

5.4

φ

1.6

1

QCD:	
  
q,	
  g	
  

5.2
5

1.4

4.8

1.2

4.6

1
−0.2

0

(c)

(a)

QCD	
  jet	
  

5.6

1.8

q	
  

0.8

Boosted QCD Jet, R = 0.6

Boosted W Jet, R = 0.6

q	
  

0.6

(b)

2.2

φ

Boson:	
  
h,	
  W,	
  Z	
  

0.4
η

0.2

0.4
η

0.6

(b)

0.8

1

−1.2

−1

−0.8

−0.6

−0.4

−0.2

η

(d)

• Can weuse in internal structure of a jet (i.e. the individual
Figure 1: Left: Schematic of the fully hadronic decay sequences in (a) W W and (c) dijet QCD
events. Whereas a W jet is typically composed of two distinct lobes of energy, a QCD jet acquires
energydepositions)
to classify
kinds
invariantmass through multiple splittings.
Right: Typicaldifferent
event displays for (b)W jets and of(d) jets?
Boosted QCD Jet, R = 0.6

5.8

+

−

5.6
5.4

φ

QCD jets with invariant mass near mW . The jets are clustered with the anti-kT jet algorithm [31]5.2 the dashed line giving the approximate boundary of the jet. The marker size
using R = 0.6, with

1.2

Jet tagging using jet substructure

−0.2

0

0.2

(a)
BoostedR = 0.6
65 GeV< m <W95Jet,
GeV

W jets
W	
  
jet	
  
QCD jets

1

QCD	
  jet	
  WQCDjetsjets

5.6

0.15
5.4

φ
Relative occurence

φ

0.06

39	
  

0.8

j

0.2

1.8

0.08

0.6

Boosted
0.6 GeV
65 QCDGeV Jet,< mR<= 95

5.8

j

2

0.1

0.4
η

(b)

2.2

0.12

Relative occurence• Typical approach:
Use physics inspired variables toprovide signal / backgrounddiscrimination

1

5.2

1.6

0.1

• Typical physics inspired variables
ΔR	
  exploit differences in:• Jet mass
(c)
(d)
(a)
(b)
• N-prong structure:(a)(b)
Figure2: 1:Distributionsof of(a)and(b) τ decay
for boosted
o 1-prong (QCD)
FigureLeft: Schematictheτfully
hadronic
sequencesWin and(a) WQCD
W jets.
and (c)For
dij
imposeaninvariantmass
window
of
65
GeV
<
m
<
95
GeV
on
jets
of
R
=
0.6,
events. Whereas a W jet is typically composed of two distinct lobes of energy, a QCD jet ao 2-prong (W,Z,H)
massBythroughmultiple thesplittings.
Typicalevent
displaysfor (b) W jets
and invariant|η| < 1.3.
themselves,τ doRight:
not offerthatmuchdiscriminating
QCD	
  
jet	
  
Jet	
  
mass	
   pow
QCD
jets
with
invariant
mass
near
m
.
The
jets
are
clusteredwith
theanti-kjet algorit
objects beyond the invariant mass cut.
o 3-prong (top)using R = 0.6, with N-­‐subje7ness	  the dashed line giving the approximate boundary of the jet. The marfor each calorimeter cell is proportional to the logarithm of the particle energies in the ce• Radiation pattern:
W	
  
jet	
  two ca
cells are colored according to how the exclusive k algorithm divides the cells
intosubjets. The open square indicates the total jet direction and the open circles indicate
o Soft gluon emission
subjet directions. The discriminating variable τ /τ measures the relative alignment ofThaler	  to&	  the
	   open square.
energy along the open circles compared
arXiv:1510.05821	
  o Color flowVan	
  Tilburg	
  
1.4

0.04

4.8

1.2

0.02

0.05

4.6

1

0
0

5

−0.2

0.2

0

0.4
0.4 0.2 0.6
τ of jet η

0.6

0.8

0.8

0 −1.2
0

1 1

1

−1

0.2

−0.8

−0.6

−0.4

η
0.4
0.6
τ of jet

−0.2

0.8

2

Boosted QCD Jet, R = 0.6

5.8

1

+

2

5.6

jet

N

5.4

φ

W

5

65 GeV < m < 95 GeV

0.05

65 GeV < m < 95 GeV

j

j

T

W jetsQCD jets

4.8

0.5

2

4.6

W jets (purple/filled)
QCD jets (blue/open)

1

0.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

η

2

0.04

0.3

τ

Relative occurence

(c)

0.06

T

5.2

0.08
0.07

−

(d)
with τN ≈ 0 have all their radiationaligned with the candidate subjet directio
0.03
0.2
therefore have N (or fewer) subjets. Jets with τN " 0 have a large fraction of their
0.02−
distributed
away sequencesfrom theincandidate
directions
0.1 and therefore have at least
Figure 1: Left: Schematic of the fully
hadronic decay(a) W + W subjet
and (c)dijet QCD
0.01
Plots
of distinct
τ1 and lobes
τ2 comparing
jets andQCD jets are shown in Fig. 2.
events. Whereas a W jet is typicallysubjets.
composedof twoof energy,Wa QCD
jet acquires
00invariant mass through multiple splittings.
Right:
Typicalevent
displays
for
(b)
W
jets
and
(d)
0
0.2
0.4
0.6
0.8
1
0
0.1 W0.2bosons.
0.3
0.4
Less obvious is how best
to use τN for identifying boosted
While0.5oneττ /τ of jet
1
2 1
QCD jets with invariant mass near mjets arethat
clusteredwithwith
the anti-k
algorithm[31] likely to be a W jet, obser
W . Theexpect
naivelyan1event
smallTτ2jetwould
be moreusing R = 0.6, with the dashed line QCD
givingjet
thecanapproximateboundary
of
the
jet.
The
marker
size
(a)
(b)
τalso
min
{ΔR
,...,
ΔRk,axis−nthough} W jets arhave smallpτT2 ,k
, as
shown
in k,axis−1
Fig. 2(b).
Similarly,
N =• Motivated data compressions,inspired by understanding of whatshould be discriminating…
• We are likely losing information!

d0∑

for each calorimeter cell is proportional to the logarithm of the particle energies in the cell. The

The Jet-Image40	
  • Treat the detector as a camera: The Jet-Image– Calorimeter towers as pixels– Energy depositions as intensity• Use all available information for jet classification– Can we learn the important information for discrimination directly fromthe data? And understand what we learned?

subjets of large radius jet as focal points ! like eyes in an image
ke use of symmetries ! Center, Rotate, and Flipoduces some smearing, but huge gain in discrimination!

Image pre-processing

Pythia 8, W'→
WZ, 8, sW'=→
13WZ,
TeV s = 13 TeV
Pythia

10
1

0.5
10-1

0.5

-2

10

Not much info!
10-30

0

-4

10

10-5
-0.5
10-6

-0.5

10-7
-1

-8-1

10

-9

-1

-0.5

0

0.5

1

[Translated] Pseudorapidity (η)

10

102
10
1

0.5

-1

10

-2

10
0

Much better!
10-3

Use subjets to align1010-0.5
images.
Make use of10
10
symmetries:-1
10
center,rotate,translate
-1
-0.5
01
0.510
1
-1
-0.5
0
0.5
-4
-5
-6
-7
-8
-9

[Translated][Translated] Pseudorapidity(η)Pseudorapidity (η)

102

102 1

T

1

T

T

102 1

103

T

Pixelate à Translateà
Average of
Average ofunrotated W jetRotate à Re-gridrotatedàWFlip
jet

103

Pixel p [GeV]103

T[Translated] Azimuthal Angle (φ)

Ts = 13 TeV

240 < p /GeV < 260 GeV, 65 < mass/GeV < 95

Pixel p [GeV][Translated] Azimuthal Angle (φ)[Translated] Azimuthal Angle (φ)

103Pixel p [GeV]

T

T

1

Pythia 8, W'→ WZ,

240 < p /GeV < 260 GeV, 240
65 <<mass/GeV
95GeV, 65 < mass/GeV < 95
p /GeV < <260

10
1

10
1

0.5

-1

10-1

-2

10

10-2

10-3 0

10-3

-4

10

10-4

10-5
-0.5
10-6

10-5

10-7

10-7

10

10-6

-8 -1

10-8

10

-9

10

Pixel p [GeV]s = 13 TeV

240 < p /GeV < 260 GeV, 65 < mass/GeV < 95[Translated] Azimuthal Angle (φ)

Pythia 8, W'→ WZ,

41	
  

-1

-0.5

0

0.5

1

[Translated] Pseudorapidity (η)

10-9

13"

Pre-processing and space-time symmetries
Pythia 8,• Translations in η are
Lorentz boosts along z-axis

s = 13 TeV

240 < pT/GeV < 260 GeV, 65 < mass/GeV < 95

Normalized to Unity

Pre-processing steps
may not be Lorentz
Invariant

42	
  

0.3

m 2J = ∑ Ei E j (1− cos(θ ij ))i< jNo pixelation
Only pixelation

0.25

Pix+Translate (naive) (×0.75)
Pix+Translate
Pix+Translate+Flip

0.2

Pix+Translate+π/2 Rotation
Pix+Translate+p2 norm (×170)– Do not preserve the pixel
energies– Use transverse energy
rather than energy as pixel
intensity

0.15• Jet mass is not invariant
under Image normalization

0

0.1

T

Naïve	
  TranslaKon	  Image	  normalizaKon	
  

0.05

60

70

80

90

100

110
Mass

Pre-processing and space-time symmetries
bright	
  	
  
side	
  

dark	
  
side	
  

Uniform	
  intensity	
  
	
  

In	
  both	  pictures	
  the	
  total	
  intensity	
  of	
  Einstein’s	
  face	  is	
  about	  the	
  same.	
  However,	
  the	
  image	
  mass	
  is	
  diﬀerent!	
  	
  hBp://mentalﬂoss.com/arKcle/49222/11-­‐unserious-­‐photos-­‐albert-­‐einstein	
  

43	
  

Pre-processing and space-time symmetries
bright	
  	
  
side	
  

dark	
  
side	
  

Uniform	
  intensity	  	  Standard	
  computer	
  vision	
  tasks	
  would	
  likely	
  not	
  want	  to	
  be	
  sensiKve	
  to	
  this!	
  
	
  
In	
  jet-­‐tagging,	
  these	  diﬀerences	
  can	  have	  physical	  meaning	
  
	
  
Need	
  physics-­‐domain	  for	
  intensity	
  of	
  
In	
  kbnowledge	
  oth	
  pictures	
  input	  the	
  total	
  
how	
  to	
  pre-­‐process	
   Einstein’s	
  face	  is	
  about	  the	
  same.	
  However,	
  the	
  image	
  mass	
  is	
  diﬀerent!	
  	
  hBp://mentalﬂoss.com/arKcle/49222/11-­‐unserious-­‐photos-­‐albert-­‐einstein	
  

44	
  

Discriminating Signal and Background45	
  Fisher• First exploredlinearDiscriminant
hBp://arxiv.org/abs/1407.5675	
  	
  Take Fisherclassification•techniques
2.5
solution and plot
QCD-like
it’s components on
applied to Jet-Images
Radiation around 1
W-like0.6

stimage grid

1.5

Q1

– Similar / improved• Fisher-Jet!
performance over physicsinspired variables• Used for

subjet in QCD jets

2.0

discrimination– Image paradigm
allows• Can be visualized
excellent insighttheto
and into
exploredunderstand physics“physics” governing
discrimination!
discriminationofthroughvisualization

1.0

0.4
Radiation along
direction
between subjet
in W-jets

No info inpresences of
1st subjet

0.5

0.0

• Linear methods can be limited– All the physics inside of a jet is not linear

0.5

0.0
0.2

Wide 2ndsubjet in
QCD jets

Hard 2ndsubjet in
W-jets0.0

0.2

0.6 < Subjet R < 0.8

1.0

1.5

2.0

Q2

Discriminant	
  f(x)	
  =	
  w*x	  2.5 Cell

0.4
0.6
0.8
1.0

Coefficient18"Jet ETmiss
ericoverviewslide
more
Going Deep
Deepnon-linearity:
Jets – Convolutional
Neural Networks
Convolution

Max-Pool

Convolution

Flatten

Max-Pool

ge

46	
  

Boosted Boson Type Tagging

Convolved
Benjamin Nachman
andAriel Schartzman
FeatureLayersConvolutions
Jet
ETmissSLAC, Stanford University

March 26,
Benjamin Nachman
and2014
Ariel Schartzman
SLAC, Stanford University

March 26, 2014 Max-Pooling
W’→ WZ event

Nachman (SLAC)

Boosted Boson Type Tagging

March 26, 20

RepeatRepeatApply deep learning techniques on jet images! [3]hBps://arxiv.org/abs/1511.05190	
  	
  Performance with Deep Neural Networks
Pythia 8,

s = 13 TeV

250 < p /GeV < 300 GeV, 65 < mass/GeV < 95

1/(Background Efficiency)

T

150

mass
τ21
ΔR
FisherMaxout100

Deep	
  NN’s	  ConvnetRandom50

State	  of	
  the	
  art	  In	
  physics	
  

0
0.2

0.4

0.6

0.8
Signal Efficiency

47	
  Combining Deep NN’s with Substructure Variables
Pythia 8,48	
  s = 13 TeV

250 < p /GeV < 300 GeV, 65 < mass/GeV < 95

1/(Background Efficiency)

T

Combine	  Deep	
  NN’s	  with	
  jet	
  mass	  150
mass+Convnet
τ21+Convnet

Combine	  Deep	
  NN’s	  with	
  substructure	
  	
  features	
  ΔR+Convnet
Maxout100

ConvnetRandomLarge	
  gains	
  from	  	
  combining	
  mass	
  with	
  
Deep	
  NN	
  
	
  
Deep	
  NN’s	  not	
  fully	
  learning	
  jet	
  mass?	
  

50

0
0.2

0.4

0.6

0.8
Signal Efficiency

Conditional Correlations with Network Output

49	
  Extracting the Physics

Looking “into” the network to better see
what it is learning50	
  d background image after applying the learned convolutional filters (right). Th-visualization
techniquehelps isunderstandnetwork learns.
Advantageof CNN
that we what
can thevisualize
the filters
Convolved
representations

Boosted Boson Type Tagging
Benjamin Nachman
and Ariel Schartzman
Jet ETmiss

51	
  SLAC, Stanford University

March 26,
Benjamin Nachman
and2014
Ariel Schartzman
SLAC, Stanford University
2DMarch
26, 2014
Convolutions
B. Nachman (SLAC)

to JetImagesBoostedBoson
Type Tagging

First	
  layer	
  11x11	  convoluKonal	
  ﬁlters	  B. Nachman (SLAC)

March 26, 2014

1 / 21

Convolved	
  jet	
  image	
  diﬀerences	
  Boosted Boson Type Tagging

March 26, 2014

Xsig*w – Xbkg*w

1/2

Average Most Activating Jet Images52	
  Physics in deep representations
signal-­‐like	
  background-­‐like	
  Pearson	  CorrelaKon	  Coeﬃcient	
  of	
  the	
  pixels	
  intensity	
  with	  the	  network	
  output:	
  how	  discriminaKng	
  informaKon	
  is	
  contained	
  within	
  the	  network	
  	
  
	
  

53	
  

Physics in deep representations

AddiHonal	
  radiaHon	
  in	
  
QCD	
  jets	  54	
  signal-­‐like	
  background-­‐like	
  SoF	
  QCD	
  gluon	
  emission	
  Pearson	  CorrelaKon	  Coeﬃcient	
  of	
  the	
  pixels	
  intensity	
  with	  the	  network	
  output:	
  how	  discriminaKng	
  informaKon	
  is	
  contained	
  within	
  the	  network	
  	
  
	
  

Learning About Learning55	
  Restricted
Space: 79 < m < 81 Pythia
GeV and< τ21s =<0.21
8, W'0.19→ WZ,
13 TeV
Pythia 8, sPhase= 13 TeV
240 < p /GeV < 260 GeV, 0.19 < τ21 < 0.21, 79 < mass/GeV < 81

240 < p /GeV < 260 GeV, 0.19 < τ21 < 0.21, 79 < mass/GeV < 81[Translated] Azimuthal Angle (φ)

1/(Background Efficiency)

20

mass
τ21

15

ΔR
MaxOutConvnet105InformaHon	
  beyond	  m,τ21	  	
  Random1

Correlation Image

0.6
0.4

0.5
0.2
0

0
-0.2

-0.5
-0.4
-1

0
0.2

0.4

0.6

0.8
Signal Efficiency

-0.6
-1

-0.5

0

0.5

1

[Translated] Pseudorapidity (η)

Learning something beyond mass and τ21 …
Spatial information indicative of radiation pattern for W and QCD:
New information learned by the network potentially related to colorflow

Pearson Correlation CoefficientTTWhere is DL in HEP going next?
• Computer vision and imaging techniques may have broad
applicability…
– Calorimeter shower classification?
– Energy calibration regression?
– Pileup reduction?
– Tracking?

• Sequential learning techniques (not discussed in this talk)may be useful in tasks with variable length data
– Typical neural networks and BDT’s require a fixed input size
– But not all discrimination tasks in HEP have a fixed size data
representation, e.g. jets with variable numbers of constituents,
variable number of jets in an events, …

• New network training paradigms may help fast
simulations, or reduce systematic uncertainties…

56	
  Dealing with Systematic Uncertainties

57	
  • Systematic uncertainties encapsulate our incomplete
knowledge of physical processes and detectors– Systematic uncertainty encoded as nuisance parameters, Z• Can we teach a classifier to be robust to these kinds of
uncertainties?

arXiv:1609.00607	
  Adversarial Networks• Adversarial training: a mini-max game– Train one neural network (f) to performthe classification task
– Train a second network (r) to predict thenuisance parameter Z from f• The loss encodes the performance ofboth classifiers, but is penalized when rdoes wellG.	
  Louppe,	
  M.	
  K.,	  K.	
  Cranmer,	  	  arXiv:1611.01046	
  	
  

58	
  

Learning to Pivot: Toy Example• 2D example• Without adversary(top) large variations
in network output
with nuisanceparameter• With adversary(bottom) performanceis independent!

G.	
  Louppe,	
  M.	
  K.,	  K.	
  Cranmer,	  	  arXiv:1611.01046	
  	
  

59	
  

Learning to Pivot: Physics Example• Tune the classification vsrobustness in training tomaximize significance, evenbeyond standard approaches• Example:– W-tagging vs QCD– Physics inspired variables as
inputs– Systematic: noise fromadditional “pileup” interactions
in collision– Count events passing minimum
network output threshold→ compute significanceincluding uncertainty (AMS)G.	
  Louppe,	
  M.	
  K.,	  K.	
  Cranmer,	  	  arXiv:1611.01046	
  	
  

60	
  

OpKmal	  tradeoﬀ	
  of	
  	
  
performance	
  vs.	
  robustness	
  Non-­‐Adversarial	
  training	
  Conclusion• Machine learning already used widely in HEP• Deep learning is a new and powerful paradigm formachine learning in certain contexts• Framing HEP data in the new ways can allow us tobenefit from deep learning• Already seen performance improvements and new
insights when using deep learning in HEP• Large potential for new image recognition and deeplearning applications in HEP61	  backup62	
  Machine Learning in High Energy Physics•

Many recent application of ML in HEP
rely on Ensembles of decision trees, suchas Boosted Decision Trees and Random
Forests•

Powerful algorithms that are relatively
simple, easy to train, and tend not to
overfit (especially Random Forests)•

They are very popular in general:– Test 179 classifiers (no deep neural networks) on 121 datasets
http://jmlr.csail.mit.edu/papers/volume15/delgado14a/delgado14a.pdf– The classifiers most likely to be the bests are the random forest (RF) versions, thebest of which (…) achieves 94.1% of the maximum accuracy overcoming 90% in
the 84.3% of the data sets•

But, Deep Neural Networks have outperformed such algorithms in certain
domains, like Object Recognition in images63	  CMS h→γγ (8 TeV)64	  

Supervised Learning
FuncKon	
  with	  adjustable	  parameters	
  

True	
  labels:	
  Higgs	
  =	
  1	
  
Bkg	
  =	
  0	
  

• Design function with adjustable parameters• Use a labeled training-set to compute error• Adjust parameters to reduce error function• Repeat until parameters stabilize• Estimate final performance on test-set

65	
  Error/Loss	  FuncKon	  Error	
  Y. Le CunClassification
x2	
  

66	
  

y=0	
  

x2	
  

y=1	
  

Rectangular	
  cuts	
  

x2	
  

y=0	
  

Linear	  discriminant	  • Learn a function to separate
different classes of data

y=0	
  

y=1	
  

y=1	
  
x1	
  

H. Voss

x1	
  

x1	
  
Nonlinear	
  discriminant	
  
x2	
  

y=0	
  

• Avoid over-fitting:– Learning too fined details about
your training sample that will notgeneralize to unseen data

y=1	
  
x1	
  

Training Improvements67	
  • Gradient descent is computationallycostly (since we compute gradient w 	
  over full training set)2

• Stochastic gradient descent– Compute gradient on one event at atime (in practice a small batch)– Noisy estimates average out– Stochastic behavior can allow “jumping”
out of bad critical points– Scales well with dataset and model size
– But can have some convergencedifficulties

w1	
  

w2	
  

– Improvements include:
Momentum, RMSprop, AdaGrad, …
w1	
  Better Activation Functions• Vanishing gradient problem– Derivative of sigmoid:∂σ (x)= σ (x)(1− σ (x))∂x

– Nearly 0 when x is far from 0!
– Gradient descent impossible!

68	
  • Rectified Linear Unit (ReLU)– ReLU(x) = max{0, x}– Derivative is constant!
∂Re LU(x) "$ 1
=#
∂x
$% 0

when x > 0
otherwise– ReLU gradient doesn’t vanish

Better Regularization Inside the Network• Dropout– Randomly remove nodes during training– Avoid co-adaptation of nodes– Essentially a large model averaging procedure69	  Deep MaxOut Networks

70	
  

MAX	
  
MAX	
  

image

MAX	
  
MAX	
  
MAX	
  
MAX	
  

MAX	
  
MAX	
  

MAX	
  

Only	
  use	
  maximum	
  of	
  	
  
several	
  neuron	
  acKvaKons	  NOT Simple Feedforward Neural Networks

71	
  • NN’s as a complex graph– Nodes of graph are the layers– Edges of graph are data flow
– Layers added to achieve a specifictask, e.g. regularization• Better to ask:

– What does each layer / module do?
– How is it connected to the previous
and next layer?

GoogLeNet	  ILSVRC	  2014	
  Winner	
  4M	
  parameters	  Incep#on	
  module	  “Network-­‐in-­‐network”	
  Weight Initializations and Training Procedures• Used to set weights to some small
initial value– Creates an almost linear classifier• Now initialize such that node outputs
are normally distributed• Pre-training with auto-encoder– Network reproduces the inputs– Hidden layer is a non-linear
dimensionality reduction– Learn important features of the input– Not as common anymore, except incertain circumstances…

• Adversarial training, invented 2014
– Will potential HEP applications later72	  ReLU Networks

hBp://www.jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf	
  	
  • Sparse propagation of activations and gradients in a network of rectifier
units. The input selects a subset of active neurons and computation islinear in this subset.
• Model is “linear-by-parts”, and can thus be seen as an exponential
number of linear models that share parameters• Non-linearity in model comes from path selection73	
  Max Pooling• Down-sample the input by taking MAX or
average over a region of inputs– Keep only the most useful information74	  Deep learning W Tagging
2.	
  CombinaKon	
  of	
  physics	
  variables	
  

	
  

75	
  

Pythia 8,

s = 13 TeV

250 < p /GeV < 300 GeV, 65 < mass/GeV < 95

Deep	
  neural	  networks:	
  x2	
  beOer	  rejecHon	
  than	  n-­‐subje7ness	
  	
  n-­‐subje7ness	
  +	
  jet	
  mass	  	
  1/(Background Efficiency)

T

150

mass+τ21
mass+Δ R
τ21+Δ R
MaxoutConvnetRandom

100

50

0
0.2

0.4

0.6

0.8
Signal Efficiency

Learned information beyond mass and n-subjettinessConvNet W tagging

76	
  Pythia 8,

s = 13 TeV

250 < p /GeV < 300 GeV, 65 < mass/GeV < 95

ClassiﬁcaHon	
  performance	
  increases	
  when	  images	
  are	  normalized	
  Since it is difficult for these
networks to learn the jet
mass, the lack of massinformation from preprocessing normalization
does not lead to worse
discrimination, while
normalization improves
training

1/(Background Efficiency)

T

150
mass

Normalized	  	
  images	  τ21
ΔRFisherMaxOut100

ConvnetConvnet-norm
Random

50

0
0.2

0.4

0.6

0.8
Signal Efficiency

units. A nonlinear unit is a transformation h✓ (x) which
produces a single value where ✓ is a vector of weights.
The most modern practice is P
to use a rectified linear
unit (ReLU): h✓ (x) = max(0, i=0 ✓i xi ). The ✓ value
hBp://arxiv.org/abs/1601.00978	
  	
   in order toat this layer willbe altered during training
minimize classification error.

Deep Neural Networks in Science77	  hBp://arxiv.org/abs/1508.04843	
  	  [)LOWHU[)LOWHU











2

F &RQYROXWLRQDO
/D\HUV [D ,QSXW,PDJH[E &RQYROXWLRQDO
/D\HUV [H2XWSXWV

G2XWSXWV

Figure 2: Crater Convolutional Neural Network (CNN)architecture computation graph. Each layer is identified
with a letter and lines show processing from left to right.
We combine the convolutional layers with fully
connected layers in a specific configration that has
achieved good performance on crater detection in Figure
2. Layer (a) is a 15x15 input image. Each candidate
exampleis craterscaledand
to non-crater
this size. Layers(b)areand(c) areFigure3: A
candidateprocessed
convolutional
layers with 20layer.
filters eachsize 4x4.
by
the first convolutional
Eight offilters
with
Each
filterispassedoverthefilterinasliding
window
interesting activation patterns are shown to the right offashionwith a stride
pixel. color.
UnlikeValues
similarare
networks
each
candidate
imageofin1 false
scaled
we
don’t
use
a
poolinglayer
asit
achievedpoor
results.
and then colored to make these values visible and to
Layer (d) contrast
is a fullywithinconnected
where
each
element
maximizeeachlayersquare
with
blue
= low
of
this
layer
is
the
result
of
h
(x)
with
unique✓sfor✓and red = high.
each element. Layer (e) has just two outputs; each

Deep NN’s for finding
craters on Mars

RegionWest (1 24+1 25)
Center (2 24+2 25)
East (3 24+3 25)

0‘09 eira ‘1 ‘11
h
d
ac
g
N
Urb
Ban
Din
CN
67.89 85.33 83.98 88.78
69.62 79.35 83.02 88.81
79.77 86.09 89.51 90.29

Table 1: F1-score via 10-fold cross validation.

Deep NN’s determining
boundaries between
neurons in the brain

Deep NNs in HEP analysis

Nature	  CommunicaKons	
  5,	
  4308	  (2014)	
  Phys.	
  Rev.	
  LeB.	
  114,	
  111801	
  (2015)	
  • Compare dense Deep NN against BDT’s andshallow NN’s
Higgs→ττ benchmark

BSM Higgs benchmark

Signal	  Background	
  

78	
  

Beyond physics inspired variables• Physics inspired variables are motivated datacompressions, inspired by our understanding ofwhat should be discriminating• We are likely losing information…
– Do we know our set of physics variables are a
sufficient over the high dimensional space of jet
physics?

• Try to make use of all information in the jet, andlearning the important information from low level
inputs79	  The ATLAS Experiment80	  Reconstructing Events81	
  neutrino	
  boBom	
  quark	
  jet	
  electron	
  

W	
  
top	
  

proton	  proton	
  

top	
  
W	
  
boBom	
  quark	  	
  jet	  up	
  and	
  down	  	
  quark	
  jets	  Daya Bay Neutrino ExperimentarXiv:1601.07621	
  

82	
  

• Aim to reconstruct inverse β-decay interactions fromscintillation light recorded in 8x24 PMT’s• Study discrimination power using CNN’s– Supervised learning → observed excellent performance (97%
accuracy)– Unsupervised learning: ML learns itself what is interesting!

Reconstructed	
  inputs	  Nonlinear	
  decoing	
  layers	
  (using	  deconvoluKons)	
  10D	
  encoding	  Nonlinear	  encoding	
  layers	
  (using	  convoluKons)	
  Inputs	  (8x24)	
  2D	
  distant	
  preserving	
  representaKon	
  of	
  
10D	
  encoding	  of	
  events