Data Reconstruction
Using

Deep Neural NetworksforLiquid Argon Time Projection Chambers(LArTPCs)
Kazuhiro Terao
SLAC National Accelerator Laboratory1Outline
1. Neutrino Physics & LArTPCs
2. Challenges: Data Reconstruction
3. Deep Neural Network Applications
4. Study on Network Response on Data
5. Future Outlook & Summary2

Neutrino Physics&
LArTPCs

3

Why Neutrinos Interesting? (I)Standard Model (SM)
Successful description of how weknow particles interact in nature… but not so much on neutrinos!
νe

νµ

ντ

Neutrinos beyond SM
With neutrino oscillations firmly in place, we know at least there
are 3 mass eigenstates. But there is much more to learn…

ν3

?
ν1

Mass hierarchy
m1 > m3?

CP violation
4

νe

νµ

ντ

ν?

Sterile neutrino?

Why Neutrinos Interesting? (II)Neutrinos are everywhere
Which makes them natural probes to the universe and its history
Relic Neutrinos

Sun

Accelerator

SuperNova

Atmospheric

Reactor

AGN

Earth

Good Stuff

EPJ H37 (2012) 3:515-565

Need to understand more about them!
Oscillation physics has taught us a lot, but still much to learn…
5

LArTPC: Particle Imaging Detector~mm/pixel spatial resolution~MeV level sensitivity

νµMicroBooNE~87 ton (school bus size)

Bubble Chamber

Liquid Argon Time Projection Chamber• Chamber-like images: digitized electronics readout• Calorimetric measurement + scalability to a large mass
6

LArTPC Experiments in U.S.
Accelerator-based oscillation experiments
DUNE: long baseline program (first beam expected @ 2026)• Measure mass hierarchy and CP violation (νµ→νe vs. νµ→νe)• Multiple 10-kton LArTPCs!
SBN: short baseline program (2015 ~)• Measure νµ→νe to investigate possible sterile neutrino oscillation• Employs three “large scale” (~100-tonnes) LArTPC detectors at
different baselines
Plate%A:%TPC%support%and%detector%
feedthroughs%
TPC%
Pla
te%
B
Se :%Cryrvi og
ce en
s%
ic%

T600$

ICARUS T600
~476 ton @ L = 600 m

ν

%
rane
emb 0cm)%
ner%M on%(6SS%In %Insula;Foam

SBN Programat Fermilab

Steel%outer%cryostat%and%support%structure%

MicroBooNE~87 ton @ L = 470 m

7

SBND~112 ton @ L = 110 m

νµ beam(BNB)LArTPC Experiments in U.S.

Use MicroBoone (where I work)
to describe typical analysis challenge
SBN Programat Fermilab

MicroBooNE~87 ton @ L = 470 m

8νµ beam(BNB)Challenges!
towardNeutrino Signal SearchHey!
I foundmy Ph.D!

νeprotone-

You need to automate that.
9Challenges for Neutrino Signal Search (I)What is hard?
100 cm100 cmChallenge 1: there may be lots of backgrounds
Cosmic Data : Run 6280 Event 6812 May 12th, 201610

Challenges for Neutrino Signal Search (II)What is hard?
100 cm100 cm≃ 14 cm x 14 cm≃ 200 cm

55 cm

Run 3469 Event 53223, October 21 st, 2015

Challenge 2: signal may vary in size (cm ~ meters)
Cosmic Data : Run 6280 Event 6812 May 12th, 201611

Challenges for Neutrino Signal Search (III)What is hard?
Cosmics
Cosmics

p

νµ

π?

µ

Cosmics

Challenge 3: identifying particle types
Necessary for selecting neutrino interaction
55 cm

Run 3469 Event 53223, October 21 st, 201512

Challenges for Neutrino Signal Search (IV)
What is hard?

Challenge 4: ClusteringMust cluster all scattered charge
depositions to reconstruct energy,
particularly hard for EM showers

13Challenges for Neutrino Signal Search (V)What is hard?

Challenge 5: programming is not easy
Need efficient, fast pattern recognition algorithms and aframework to run a chain (or multiple chains) of them

we

sh
i
w

Not how it looks in our code
in r

eal
ity

Our data is an “image”,
a matrix of numbers

01101010100101011010101001011010
10111010101001010100010010101101
0101001011010101001010110101010
01011010101001010110101010101101
0101001010110101010010110101010
01011010101001010110101010010110
10101001010110101010101101010100
10101101010100110101101010100101

How it actually looks in our code14

… so what shall we do? …
Solutions?
• Solution A: “traditional path”- Hand-engineered reconstruction algorithms‣ Develop signal search analysis cuts

• Solution B: machine learning- Suited tool for a pattern recognition- “Deep Neural Networks”‣ In particular…
Convolutional Neural Networks (CNNs)‣ Scalable technique, generalizable to various tasks
15

NCπ0
CC1π

CCQE

DIS..!

Deep Neural Network
Applications
Outline1. Neutrino Physics & LArTPCs
2. Challenges: Data Reconstruction
3. Deep Neural Network Applications
4. Studying Network Response on Real Data
5. Summary
16

Convolutional Neural Networks
Image
Classification

Context Analysis

Pixel Classification

self-driving car,image captioning,playing a boardgame,… and more!

Image
Classification17

Traditional “Neural Net” in HEP
Fully-Connected, Feed-forward,
Multi-Layer Perceptrons

A traditional neural network consists of a stack of layers of suchneurons where each neuron is fully connected to other neurons of
the neighbor layers
18Problems Applying for Image Analysis
Fully-Connected, Feed-forward,
Multi-Layer Perceptrons

Feed in entire image
Cat?
Problem: scalability

Use pre-determined features
Cat?
Problem: generalization
19

Convolutional Neural Networks
CNN introduce a limitation by forcing the network tolook at only local, translation invariant featuresneuroninput

feature map

Activation of a neuron depends
on the element-wise product of
3D weight tensor with 3D input
data and a bias term(b) Feed-forward neural network• Translate over 2D space to process the whole input• Neuron learns translation-invariant features- Suited for a “homogeneous” detector like LArTPC• Output: a “feature-enhanced” image (feature map)
20

Convolutional Neural Networks

Toy visualization of the CNN operation
21

Convolutional Neural Networks
Introduction to CNNs

Feature Map. . ..
. .

-2 0
-1 -1
-3 -1
1 0

.

.

1 0
2 1
0 3
-1 0

1
0
2
.

2

0

1

Filter

1 0 2 11 1 1 -1
0 1 0 -1
-1 -1 -1 1
0 0 -2 0
1 -1 0 1
Image
0 1 -3 -1-1 1 -2 1

Image

weights

DL µB NPToy visualization of11 the CNN operation
Genty

DL µB NP

22

12

Convolutional Neural Networks
Feature Maps

Introduction to CNNs

Depth

N Filters

Image

many weights!
Genty

DL µB NP

apply
many filters
13

Toy visualization of the CNN operation
23

Convolutional Neural Networks
Feature map visualization example• https://www.youtube.com/watch?v=AgkfIQ4IGaM

Neuron concerning face24

Neuron loving texts
more than your faceApplications
in

Reconstruction
Outline1. Goals: What We (or I for now) Want
2. Introduction to Convolutional Neural Networks
3. Applications
4. Path Toward Goals
25

Application of CNNs
• Categorization- What’s in a picture?
- Particle ID• Detection- What in where? (bounding box)
- Find a neutrino•Semantic Segmentation- What in where? (pixel level)- Clustering!
26

Application of CNNs
• Categorization- What’s in a picture?
- Particle ID• Detection
What in where? (bounding box)
Find a neutrino• Semantic Segmentation
What in where? (pixel level)
Clustering!
27

CNN for Image Classification

Input Image

Down-sampled
Feature MapsClasses

Feature map preserves spatial
information•Goal: provide a single label for the whole image

28

CNN for Image Classification

Input Image

Down-sampled
Feature MapsClasses

Feature map preserves spatial
information•Goal: provide a single label for the whole image•How: transform the higher spatial resolution input(i.e. image) into a vector of image features, ultimately
a 1D array of feature parameters, useful for image
classification
29

CNN for Image Classification
Neutrino EventClassifier

MicroBooNE
arxiv:1611.05531

Using 2D images,
classify neutrino vs.
non-neutrino (signal
vs. background)“Siamese Tower”
to combine multipleprojection 2D images
for an event-wiseclassification task
Check our publication
for details…
30

Input “images”

CNN for Image Classification… from data reconstruction perspective …
Cons. to Image Classification• Highest level abstraction: N^2 (pixels) into 1 variable- Ultimately we just want to know a neutrino flavor & energy (2variables!)• Big black-box invite concerns- How is network making a decision?
- A way to enhance the traditional (non-ML) approach?

… this is cat!

55 cm

Run 3469 Event 53223, October 21 st, 2015

31

CNN for Image Classification… from data reconstruction perspective …
How about… CNNs for feature extraction?
• Networks to extract key features for physics analysis- Finding an interaction vertex, clustering of particle’s pixels, etc.
- Physics driven guidance + physical observables to output- … essentially “data mining”• Helps to understand…
- How/where detector systematics matters- Combining algorithmic approach where appropriate‣ Let’s not forget CNN is just a functional approximation (i.e.
geometrical transformation) of statistical distributions…

32

Application of CNNs
• Categorization- What’s in a picture?
- Particle ID• Detection
What in where? (bounding box)
Find a neutrino• Semantic Segmentation
What in where? (pixel level)
Clustering!
33

CNN for Object Detection
Faster-RCNN

Region Proposal Network
Proposes lots of (100s) possible
bounding boxes for “an object”(used in our paper)

Region Classification Network

Region Proposal Network (RPN)

Detector
Netwrok34

FC/Conv

Regress (fine-tune) further a proposed
region for pre-defined set of concrete
object classes

CNN for Object Detection
Neutrino interaction detection
Trained a network to find neutrino interaction• Training sample uses simulated neutrino + cosmic data image- Supervised training using ≃100,000 images (single plane view)

arxiv:1611.05531

νµYellow: “correct”
bounding box
Red: by the network
Network Output≃ 2.6m (width) x 1 m (height)

MicroBooNE
Simulation + Data Overlay
35

CNN for Object Detection
Neutral Pion Detection
Similar applications to collect neutral pions• Detector calibration & EM shower energy reconstruction

Red: by the network
MicroBooNE Data
Preliminary
In-Progress Analysis
36

Application of CNNs
• Categorization- What’s in a picture?
- Particle ID• Detection
What in where? (bounding box)
Find a neutrino• Semantic Segmentation
What in where? (pixel level)
Clustering!
37

Recap: Classification Network
Feature extraction by CNN“Written Texts”
feature map

After
1st convolution

Feature e

After
3rd convolution
After
2nd convolution

xtractionby CNN

After steps of down-sampling,“feature map” still preserves arough object location information
38“Human Face”
feature map

How SSNet Works
Goal: recover precise, pixel-level location of objects
1. Up-sampling- Expand spatial dimensions of feature maps

2. Convolution

Down-samplingUp-sampling

featuretensor
39

Intermediate, low-resolution
feature map

Output Image

Input Image- Smoothing (interpolation) of up-sampled feature maps

The First Application: Track/Shower Separation
Supervised Training (UB)
• Using simulation! Requires pixel-wise “labels”

Input Image

Label Image

Input Image“Label” Image(for training)

Training techniques R&D• The bottoms-up simulation in our field (HEP) provides
pixel-level simulation information, powerful for enforcing
network’s learning. See back-up slides if interested in…
40

The First Application: Track/Shower Separation
νeprotone-

MicroBooNE
Data CC1π0

ADC ImageMicroBooNE
Data CC1π041

Network Output

Network ResponseonReal Data

Outline1. Neutrino Physics & LArTPCs
2. Challenges: Data Reconstruction
3. Deep Neural Network Applications
4. Studying Network Response on Real Data
5. Summary
42

Can We Use/Trust DNN?
Typical questions…
• Is network trained on simulation generalized for data?
• Is network response (features it finds) reasonable?
• How does unknown effect (detector systematics) affect?

43

Can We Use/Trust DNN?
Typical questions…
• Is network trained on simulation generalized for data?
• Is network response (features it finds) reasonable?
• How does unknown effect (detector systematics) affect?

Rest of the slides…
Took 1st preliminary steps to address first two concerns.
The third concern not addressed in this talk, but can beaddressed by tuning simulation parameters to study how
the network response changes…
44

Benchmarking SSNet w/ Real Data
Approach (big picture)
1. Collect image samples from data & simulation
running the same traditional algorithms
2. For both data & simulation, prepare pixel-level
labels made by physicists (software tools developed)
3. Run the network on both data & simulation, andcompare the disagreement rate (pixel fractions) withhuman-made labels
Network
Input

Pixel-level
Label by Human45Benchmarking SSNet w/ Real Data
Samples (100 images per sample per sim/data)
A cosmic ray muon decay- Involves both “track” and “shower”, simple and intuitive.

Neutrino interactions- More complicated: varying particle types and multiplicitye-ExampleMuon decayµ46

MicroBooNE Data
Preliminary
In-ProgressDecay Muons: Example Displays

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Input Image

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Human Label47

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

SSNet Label4 Visually Picked “Busy Neutrino Events”
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Input Image

Human Label48

SSNet Label

Overall Performance• Data/Simulation agreement within statistical error- No systematic error included• Network does better than a human analyzer (sim.)
Muon Decay

Neutrino w/ Gamma

Disagreement rate mean/std in %Disagreement rate mean/std in %

ry
a
n
i
m
i
l
e
r
s
P
s
e
r
g
o
r
P
n
I

ry
a
n
i
m
i
l
e
r
s
P
s
e
r
g
o
r
P
n
I

Human vs. Network
Disagreement pixel
fraction per event(100 images)

Human vs. Network
Disagreement pixel
fraction per event

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress49

Decay Muons: Pixel Value Variation
Studied how network performance varies whenpixel values are scaled by a constant factor
Data
MC

Data x 0.75
MC

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

No scaling

y
r
a
n
i
m
i
s
Prel
s
e
r
g
o
r
P
InChange in the mean error rate is within 1% when50
pixel values are scaled within
20%, fairly robustDecay Muons: Inter-Pixel Correlation
Study, qualitatively, how network reacts tointeresting portions of an image
Bragg peak
Low energy
electron

MIP part of a track

MicroBooNE Data
Preliminary
In-Progress51

Decay Muon: Inter-Pixel Correlation … Region-0
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

52

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-ProgressDecay Muon: Inter-Pixel Correlation … Region-0
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

In region-0 (MIP muon track),
the longer the track becomes, thenetwork labels
pixels as a track
MicroBooNE Data
MicroBooNE Data
Preliminary
Preliminary
MicroBooNE Datawith higher
In-Progress
In-Progress confidence.
Preliminary
In-Progress(follows our intuition!)

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

53

MicroBooNE Data
Preliminary
In-ProgressDecay Muon: Inter-Pixel Correlation … Region-1
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

54

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-ProgressDecay Muon: Inter-Pixel Correlation … Region-1
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

In region-1 (muon Bragg peak),MicroBooNE Data
MicroBooNE Data
wherehigh/increasing
dE/dX isPreliminary
Preliminary
In-Progress
In-Progress
visible, the network is fairly
Dataconfident MicroBooNE
toPreliminary
label
pixels as track
MicroBooNE Data
In-Progressdisregard of a track length Preliminary
In-Progress(distinction from MIP part!)
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress55

Decay Muon: Inter-Pixel Correlation … Region-2
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress56

Decay Muon: Inter-Pixel Correlation … Region-2
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

In region-2 (Michel electron), the network is almost
MicroBooNE
zero-confidence
Data
when
MicroBooNEgivenDataastraight
MIP-like
Preliminary
Preliminary
In-Progress
In-Progresselectron trajectory. However
when connected withwabbling shower-like component, the straight
MicroBooNE Data
trajectory part is classified with high confidence
Preliminary as ashower. When connected with Bragg peak,In-Progress
it has a
MicroBooNE
Data preferenceMicroBooNE
Data predicting a straightslighttowardPreliminary
Preliminary
In-Progresselectron In-Progress
track as a shower.
(see clear correlation with neighboring pixels)
MicroBooNE Data
Preliminary
In-Progress
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress57

Neutrino w/ Gammas: 4 Worst Events
MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Input Image

Disagreement
MicroBooNE16.6
Data%
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Disagreement
16.6 %
58

SSNet LabelNeutrino w/ Gammas: 4 Worst Events
Disagreement
12.5 %

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

MicroBooNE Data
Preliminary
In-Progress

Input Image

Disagreement
16.2 %
59

SSNet Label… wrapping up …
Outline1. Neutrino Physics & LArTPCs
2. Challenges: Data Reconstruction
3. Deep Neural Network Applications
4. Studying Network Response on Real Data
5. Future Outlook & Summary
60

R&D for “Full” Reconstruction Chain• Ultimate goal: 3D data analysis- Need to demonstrate detections & clustering- Software development projects (visualization, data rep., IO)- Recent success: 3D “classification” working well
‣ Moving onto detection & clustering…
Muon

Network
Gamma

Reconstructed
3D View

3D Clustering
+ PID
61

R&D for “Full” Reconstruction Chain• Full Chain- Combine networks and develop end-to-end training scheme- Place to employee recursive neural networks (RNNs),
multi-network-cascade model62

Call for Collaborative Work!
Would be great to share knowledge/tools…
• Technique/Experience sharing• Shared software development- Data rep., IO & file format, visualization tools, image processing algo
‣ My favorite: C++ frameworking w/ extensive Python API, QT-based python
visualization tools… but I’d love to learn better what SLAC people use!

- Extending open source deep neural network software APIs
- Deployment of software (e.g. how to use Singularity container)- … anything :) please let me know!

• Workshops to spread expertise- Educate “students” (count me), raise next generation researchers- Educate “lectures” (maybe me) for good future workshops‣ Really important to strengthen the whole lab
Recently started to organize something similar for LArTPC
community… would be nice to have one at SLAC beyond LArTPC…
63

Thank you!
for your attention :)Take Away Messages
1. Modern detectors (e.g. LArTPC) need advanced patternrecognition algorithms
2. Deep neural networks can be a reconstruction tool as
physics feature extractor
3. Validation with real data is important to convince
ourselves, and we are taking initiatives to this problem
64Implementation of SSNet
for
MicroBooNE

65

How Classification Network Works
Goal: extract features to give “single label” to an image
1. Convolution operation- Creates spatial “feature map”

2. Down-sampling- Turn an image into an array of discriminants

Input Image

neuroninput

Down-sampling
Feature Maps

DiscriminantsAfter
1st convolution

feature map(b) Feed-forward neural network

Convolution Operation
66

After
3rd convolution
After
2nd convolution

Series of convolutions+ down-samplingHow Classification Network Works
Feature extraction by CNN“Written Texts”
feature map

After
1st convolution

Feature e

After
3rd convolution
After
2nd convolution

xtractionby CNN

After steps of down-sampling,“feature map” still preserves arough object location information
67“Human Face”
feature map

How SSNet Works
Goal: recover precise, pixel-level location of objects
1. Up-sampling- Expand spatial dimensions of feature maps

2. Convolution

Down-samplingUp-sampling

featuretensor68

Intermediate, low-resolution
feature map

Output Image

Input Image- Smoothing (interpolation) of up-sampled feature maps

U-ResNet: Architecture
Shower vs. Track
Pixel Categorization
Network(single plane!)69

U-ResNet: Architecture70

U-ResNet: Architecture71

U-ResNet:
U-ResNet
Architecture
p
µ

γ
p

µ

output“track” prediction
512 x 512γIntermediate
Feature Maps
(Picked 1 feature map
for which track pixels’
activation is high)input image
512 x 512
256 x 256128 x 128

64 x 64

32 x 3272

U-ResNet: Architecture“Concatenation”
of feature maps before down-sampling to
the up-sampled ones of the same size tohelp more accurate interpolation operations(U-Net links: paper and video)73

U-ResNet: Architecture

Use ResNet (bottleneck) modules inside
U-Net architecture74

U-ResNet: Output
Input Image(zoomed to vertex)

νe

ep

75

U-ResNet: Output
Output Image(zoomed to vertex)

Input Image

Background(zoomed to vertex)

νeeShower

p

Track

76

U-ResNet: Output
Output Image(zoomed to vertex)

Input Image

Background(zoomed to vertex)

νeeShowerp

TrackAnalysis Interpretation(Categorize by pixel-score)77

How to Train SSNet: Big Picture
Supervised Training• Use “labeled image” = answer (background, track, shower)• Define pixel-wise “loss”- Can be only 1 of 3 categories (multinominal logistic regression)

Input Image“Label” Image
Kronecker delta where ci is(for training)

the true category of i-th pixel

where
78

Probability of i-th
pixel belonging to
the j-th category

Training Techniques (I)1. Transfer Learning- Idea to use pre-trained filters (weights)- Trained the first half of the network for 5-particle
classification with data set used for the 1st paper.

[e-, γ, µ-, π-, p]79

Training Techniques (II)
1. Transfer Learning- Idea to use pre-trained filters (weights)- Trained the first half of the network for 5-particle
classification with data set used for the 1st paper.

2. Class-wise Loss (CL) Weighting- LArTPC Images mainly backgrounds (>99%)- Up-weight the network’s mistake (loss) with adifferent factor per category‣ 1:100 for background:shower/track

Fraction of pixels
that carry charge
above 10 pixel value
80 (≃1/4 MIP) in percent

Training Techniques (III)
3. Pixel-wise Loss (PL) Weighting- Beyond class-wise: assign weight per pixel- Pixel weight = inverse of pixel count in the category- 4 categorization scheme studied

Category definitions

81

Category definitions

MicroBooNE
How It Works82

MicroBooNE: Introduction
MicroBooNE
TPC & Cryostat

Drawing courtesy of Adrien H.

BNB• Short baseline (1m/MeV)νµ ⇒ νe oscillation• Booster neutrino beam
- neutrino energy O (1 GeV)νµ“Design and Constructionof MicroBooNE detector” (JINST 12, P02017)

νe• LArTPC detector- 90 tonnes TPC active volume

83

TPC Working Principle (I)X = 2.5 m

1. Charged particles interact in Ar• Ionize argon• Produce scintillation light
2. Ionization e- drift toward anode
3. Wire planes detect drift e-

Z

=

10

.4

m

Y = 2.3 m

νCathode @ 70 kV(plate)Electric Field~270 V/cm

Anode(wire plane)84

TPC Working Principle (II)X = 2.5 m

Y = 2.3 m

1. Charged particles interact in Ar• Ionize argon• Produce scintillation light
2. Ionization e- drift toward anode
3. Wire planes detect drift e-

Electrons

Z

=

10

.4

m

Scintillation Light

Scintillation Lightdetected by PMTs

Cathode @ 70 kV(plate)Electric Field~270 V/cm

Anode(wire plane)85

TPC Working Principle (III)X = 2.5 m

Y = 2.3 m

1. Charged particles interact in Ar• Ionize argon• Produce scintillation light
2. Ionization e- drift toward anode
3. Wire planes detect drift e-

Drift Time = X position

Z

=

10

.4

m

Three
Wire Planes

Charge collectedby wire plane

86

Cathode @ 70 kV(plate)Electric Field~270 V/cm

Anode(wire plane)

Scintillation Lightdetected by PMTs

Wire Plane Projection (2D)
T=t0

µ
e

Wires

87

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e

Wire Plane Projection (2D)
T=t0

µ

e

Ddrift1

e

e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e

Wires

Amplitude

Wire1 detects @ T=t1
Ddrift1 / Vdrift = (t1 - t0)

Time
t0

88

t1

Wire Plane Projection (2D)

Ddrift1

e

e
e

e
e

Ddrift2

µ

e
e

e
e

e
e

e
e

e
e

Amplitude

T=t0

e
e

e
e

e

Time

e

e
t0

t1 t2

Wire2 detects @ T=t2
Ddrift2 / Vdrift = (t1 - t0)
Wires

Amplitude

Wire1 detects @ T=t1
Ddrift1 / Vdrift = (t1 - t0)

Time
t0

89

t1

Wire Plane Projection (2D)
…putting together wires…

µ

e

Ddrift1

e

e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e

Amplitude

T=t0

Time

voila!

Wires

Amplitude

Wire1 detects @ T=t1
Ddrift1 / Vdrift = (t1 - t0)

Time
t0

90

t1

Wire Plane Projection (2D)
T=t0

µ

e

Ddrift1

e

e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e
e

e

Induction

3 WirePlanesAmplitude

Collection
Wire1 detects @ T=t1
Ddrift1 / Vdrift = (t1 - t0)

Time

91

t0

t1

MicroBooNE TPC & Cryostat

Anode Wire Plane

Cathode Plate92

MicroBooNE TPC & Cryostat

Anode Wire Plane

Cathode Plate93

MicroBooNE TPC & Cryostat

94

Images From Our Detector

100 cm100 cm

Cosmic Data : Run 6280 Event 6804 May 12th, 201695

Images From Our Detector

CosmicsνµCosmics

55 cm

Run 3469 Event 53223, October 21 st, 2015
96

NCπ0
CC1π

CCQE

DIS..!

Convolutional
Neural Networks
How Does It Work?
97

Image Analysis: Identifying a Cat

98

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat

A cat = collection of certain shapes(object modeling in early days)
99

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat

A cat = collection of certain shapes(object modeling in early days)
100

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat… how about this?

Take into account for a view point
101

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat… how about this?

… and maybe more shapes
102

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat… gets way worse …

… I (a human) am never taught exactly how
cat should look like by anyone, but I somehow
can recognize them really well.
103

Taken from slides by Fei-Fei’s TED talkImage Analysis: Identifying a Cat… gets way worse …

A breakthrough: a machine learning algorithm
that forms (trains) itself by sampling a large setof data to “learn” how cat looks like (distribution)
104

Taken from slides by Fei-Fei’s TED talkIntroduction to CNNs (I)ImageClassification

Context Analysis

Pixel Classification

self-driving car,image captioning,playing a boardgame,… and more!

Image
Classification
105

Introduction to CNNs (II)Background: Neural Net⟶

x[

Takes in a vector of inputs (x).
Commonly inputs are summed
with weights (w) and offset (b)then run through activation.

x0
x1

w0

⋮

⋮

∑

xn

wn

+b

σ( x➞)

NeuronSum

Activation
Outputw1[

The basic unit of a neural net
is the perceptron (loosely
based on a real neuron)

Input

106

Introduction to CNNs (II)
Perceptron 2D Classification
Imagine using two features to separate cats and dogs
∑00
from wikipedia

∑0

cat
dog

[

x1

[

x0

OutputBy picking a value for w and b,
we define a boundary
between the two sets of data
107

Introduction to CNNs (II)
Perceptron 2D Classification
Maybe we need to do better: assume new data point(My friend’s dog — small but not as well behaved)∑0(Thor)

0from wikipedia
108

∑0

cat
dog

[

x1

[

x0

OutputIntroduction to CNNs (II)
Perceptron 2D Classification
Maybe we need to do better: assume new data point(My friend’s dog — small but not as well behaved)∑0∑1(Thor)

0from wikipedia

x0

∑0

x1

∑1

We can add another perceptron
to help (but does not yet solve
the problem)
109

Introduction to CNNs (II)
Perceptron 2D Classification
Maybe we need to do better: assume new data point(My friend’s dog — small but not as well behaved)(Thor)

∑0

∑0

x1

∑1

∑2

cat
dog

[

∑2

x0

[

∑1

OutputAnother layer can classify based on
preceding feature layer output
110Introduction to CNNs (III)“Traditional neural net” in HEP
Fully-Connected Multi-Layer Perceptrons

A traditional neural network consists of a stack of layers of suchneurons where each neuron is fully connected to other neurons of
the neighbor layers
111

Introduction to CNNs (III)“Traditional neural net” in HEP
Problems with it…

Feed in entire image
Cat?
Problem: scalability

Use pre-determined features
Cat?
Problem: generalization
112

Introduction to CNNs (III)CNN introduce a limitation by forcing the network tolook at only local, translation invariant featuresneuroninput

feature map

Activation of a neuron depends
on the element-wise product of
3D weight tensor with 3D input
data and a bias term(b) Feed-forward neural network• Translate over 2D space to process the whole input• Neuron learns translation-invariant features• Applicable for a “homogeneous” detector like LArTPC
113

Want more details?
Feel free to ask me later!

Convolutional Neural Networks

Toy visualization of the CNN operation
114

Convolutional Neural Networks
Introduction to CNNs

Feature Map. . ..
. .

-2 0
-1 -1
-3 -1
1 0

.

.

1 0
2 1
0 3
-1 0

1
0
2
.

2

0

1

Filter

1 0 2 11 1 1 -1
0 1 0 -1
-1 -1 -1 1
0 0 -2 0
1 -1 0 1
Image
0 1 -3 -1-1 1 -2 1

Image

weights

DL µB NPToy visualization of11 the CNN operation
Genty

DL µB NP

115

12

Convolutional Neural Networks
Feature Maps

Introduction to CNNs

Depth

N Filters

Image

many weights!
Genty

DL µB NP

apply
many filters
13

Toy visualization of the CNN operation
116

Neutrino Oscillation
Experiments~ Detectors ~117

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Inverse Beta Decay (IBD)νe + p → e+ + n
by Reines & Cowan (Nobel Prize 1995)

Cd-doped waterFirst neutrino detection
0.4 ton, 100 PMTs
118

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Birth of neutrino
astrophysics!
KamiokaNDE Detector
3 kton ultra-pure water, 1000 20” PMTs
(shared Nobel Prize 2002)
119

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Ultra-pure water
3 kton, 1000 PMTs
(1983)

Discovery ofνatmo oscillation!
Super-KamiokaNDE
50 kton ultra-pure water,
11000 PMTs
(shared Nobel Prize 2015)120

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Ultra-pure water
3 kton, 1000 PMTs
(1983)

Ultra-pure water
50 kton, 11000 PMTs
(1996)

Discovery ofνsolar oscillation!
SNO
1 kton heavy-water Cherenkov,
9600 PMTs(shared Nobel Prize 2015)121

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Ultra-pure water
3 kton, 1000 PMTs
(1983)

Ultra-pure water
50 kton, 11000 PMTs
(1996)

Reactor neutrino
oscillation!
(the solar model is right!)
KamLAND
1 kton liquid scintillator, 1900 PMTs
My first neutrino experiment(undergraduate RA @ UC Berkeley)
122

Heavy water
1 kton, 9600 PMTs
(1999)

Neutrino Oscillation Experiments (I)Eνolution of Detectors

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Ultra-pure water
3 kton, 1000 PMTs
(1983)

Ultra-pure water
50 kton, 11000 PMTs
(1996)

Heavy water
1 kton, 9600 PMTs
(1999)“Near” & “Far” design
2 x 16 ton detectors with 400
PMTs each (Double Chooz)
My Ph.D thesis! (MIT)Liquid Scintillator
1 kton, 1900 PMTs
(2002)“Last mixing angle”
Gd-doped liquid scintillator
RENO, Daya Bay, Double Chooz
123

θ13 Experiments!

Neutrino Oscillation Experiments (I)Water Cherenkov Detector
Super-Kamiokande124

Neutrino Oscillation Experiments (I)Water Cherenkov Detector
Super-Kamiokande
125Neutrino Oscillation Experiments (I)Liquid Scintillator Detector
KamLAND

Less topological information
but excellent energy resolution126Neutrino Oscillation Experiments (II)How can we do better?
Three important detector features for oscillation measurementeGood Energy
Resolution

Large Mass(scalable)

Particle ID
Capability

Precise Eν reduce
oscillation uncertainty“More” statistics to measure
rare physics process

Better ν identification
background rejection127

Neutrino Oscillation Experiments (II)νµBubble Chamber

Liquid Argon Time Projection Chamber• Digitized bubble Chamber-like images• Calorimetric measurement + scalability to a large mass128