Machine Learning for
Particle Imaging Detectors inExperimental Neutrino Physics
Kazuhiro Terao
SLAC National Accelerator Lab.
Deep learning and Physics 2020 Seminar Series
1Outline
1. Neutrino detectors
2. Machine Learning & Computer Vision Applications
3. ML-based Neutrino Data Reconstruction Chain
4. Summary2

Detectors
forNeutrino Oscillation
Experiments

3

Machine Learning & Computer Vision in Neutrino Physics
Neutrino Detectors: Early Days

Cd-doped water
0.4 ton, 100 PMTs
(1956)

Inverse Beta Decay (IBD)νe + p → e + + n
by Reines & Cowan (Nobel Prize 1995)

First neutrino detection

4

Need for advanced algorithms
for analyzing high resolution data withcomplex topologies.
(goal: maximize physics output)

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers~mm/pixel spatial resolution~MeV level sensitivity

νμMicroBooNE~87 ton (school bus size)

Liquid Argon Time Projection Chamber• High resolution photograph of charged particle trajectories• Calorimetric measurement + scalability to a large mass

8

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers

Topological shapedifference is a major
distinction for “shower”
particles

9

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers

Trajectory ends are
distinct, and useful forseeding particle
clustering and trajectory
fitting10

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers

Many, local kinks
caused by Multiple Coulomb
Scattering process can beused for momentum
estimation

11

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers

Small branches on muon-liketrajectories are knocked-off
electrons, useful key for the
direction
12

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers

Stopping
particle

Energy deposition
patterns (dE/dX)
vary with particle mass
& momentum, usefulfor analysis
e- vs. γusing dE/dX

13

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers
Do you see neutrino interaction here?
100 cm100 cm

Cosmic Data : Run 6280 Event 6812 May 12th, 201614

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers
Nope :) In this detector, only ~1/700 beam neutrino interacts
100 cm100 cm

Cosmic Data : Run 6280 Event 6812 May 12th, 201615

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers… and 1/700 have many variations in hi-resolution imaging...
100 cm100 cm

Cosmic Data : Run 6280 Event 6812 May 12th, 201616

Machine Learning & Computer Vision in Neutrino Physics
Time Projection Chambers (3D ones)

DUNE-ND: on average ~dozen neutrino interactions per event
Detector: pixelated LArTPC for 3D imaging for high overlaps

17

Machine Learning andComputer Vision18

Machine Learning & Computer Vision in Neutrino Physics
You can find a cat? You can find a neutrino!

How to write an algorithm toidentify a cat?
… very hard task ...

19

Machine Learning & Computer Vision in Neutrino Physics
You can find a cat? You can find a neutrino!

Development Workflow for non-ML reconstruction
1. Write an algorithm based on physics principles

algorithm

collection of
A cat =
(or, a neutrino)
certain shapes

20

Machine Learning & Computer Vision in Neutrino Physics
You can find a cat? You can find a neutrino!

Development Workflow for non-ML reconstruction
1. Write an algorithm based on physics principles
2. Run on simulation and data samples
3. Observe failure cases, implement fixes/heuristics
4. Iterate over 2 & 3 till a satisfactory level is achieved
5. Chain multiple algorithms as one algorithm, repeat 2, 3, and 4.

algorithm
Partial cat(escaping the detector)

Stretching cat (Nuclear FSI)

collection of
A cat =
(or, a neutrino)
certain shapes

21

Machine Learning & Computer Vision in Neutrino Physics
You can find a cat? You can find a neutrino!

Development Workflow for non-ML reconstruction
1. Write an algorithm based on physics principles
2. Run on simulation and data samples
3. Observe failure cases, implement fixes/heuristics
4. Iterate over 2 & 3 till a satisfactory level is achieved
5. Chain multiple algorithms as one algorithm, repeat 2, 3, and 4.

“Machine learning”● Automatization of step 2, 3, and 4.
● Well-defined error propagation (step 5).
● Can optimize the whole chain for physics.
Next: what kind of ML algorithms?

22

Machine Learning & Computer Vision in Neutrino Physics
My Research
Machine Learning for Data Reconstruction● Goal: high level abstract information (like image classification)

Electron
Neutrino

Input Data

High-level
Output23

Machine Learning & Computer Vision in Neutrino Physics
My Research
Machine Learning for Data Reconstruction● Goal: high level abstract information (like image classification)● How: design the algorithm = data transformation architecture thatextracts a hierarchy of physically meaningful features (evidences)
Electron
Neutrino

Multi-task
Algorithm
Input Data Pixel Feature
Extraction + Points

Pixel clustering

High-level
OutputKinematics
Inference

p
pi

e

p

… becomes
interpretable! 24

Machine Learning & Computer Vision in Neutrino Physics
My Research
Machine Learning for Data Reconstruction● Goal: high level abstract information (like image classification)● How: design the algorithm = data transformation architecture thatextracts a hierarchy of physically meaningful features
Input

Pixel Feature

Pixel Clustering

Particle Clustering

The Rest: describe the chain

25

ML-Based LArTPC
Data Reconstruction26

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility
Distinguish 2 distinct particle topologies: showers v.s. tracks
Critical to deploy different algorithms for clustering pixels in the next stage.

PRD 99 092001
arXiv:1808.07269

27

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility
Architecture: U-Net + Residual Connections
Encoder

Decoder

input
conv
conv-s2-ﬁnc
tconv-s2-fde
conv-fdec

Number of strided
convolutions, convolution
layers, residual connections,differ in impementations
Image credit: Laura Domine @ Stanford

softmax
Residualconnections
ConcatenationML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility“Applying CNN” is simple, but is it scalable for us?
CNN applies
dense matrix
operations
In photographs,
all pixels aremeaningful
grey pixels = dolphins,
blue pixels = water, etc...

29

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility“Applying CNN” is simple, but is it scalable for us?
LArTPC data is generally sparse, but locally dense
CNN applies
dense matrix
operations<1% of pixels
are non-zero in
LArTPC data

In photographs,
all pixels aremeaningful

Zero pixels aremeaningless!
grey pixels = dolphins,
blue pixels = water, etc...

Figure credit: Laura Domine @ Stanford

Empty pixels = no energy

Figures/Texts: courtesy of
Laura Domine @ Stanford30ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility“Applying CNN” is simple, but is it scalable for us?
LArTPC data is generally sparse, but locally dense
CNN applies
dense matrix
operations<1% of pixels
are non-zero in
LArTPC data

In photographs,
all pixels aremeaningful

Zero pixels aremeaningless!
Figures/Texts: courtesy of
Laura Domine @ Stanford○ Scalability for larger detectors■ Computation cost increases linearly with the volume■ But the number of non-zero pixels does not
Figure credit: Laura Domine @ Stanford

31ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility

Sparse Submanifold Convolutions32

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility

Sparse Submanifold Convolutions

33

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility
Our data is locally much more dense
than ShapeNet 3D dataset… which makes convolution filter
more effective on our data as long as
the sparsity issue is handled
Image credit: Laura Domine @ Stanford

34ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility

Sparse U-ResNet fits more data in GPU + good scalability@batch size 88
sparse uses
93x less memory
than dense andcomputation is
3x fasterCan handle easily thewhole ICARUS detector
which is x6 larger than
MicroBooNE.
DUNE-FD is piece of
cake (larger volume but
less non-zero pixels)

Figure credit: Laura Domine @ Stanford35ML-based Neutrino Data Reconstruction Chain
Stage 1-a: Pixel Feature Extraction + Scalablility
Sparse Sub-manifold
Convolutional NN

Type

ProtonMu/PiShower

Delta

Michel

Acc.

0.99

0.98

0.99

0.97

0.96

● Public LArTPC simulation○ Particle tracking (Geant4) + diffusion, no
noise, true energy● Five type segmentation

PhysRevD.102.012005 presented @ ACAT 2019

● Memory reduction ~ 1/360● Compute time ~ 1/30
● Handles large future detectors

Mu/pi
Proton
EM Shower
Delta Rays
Michel36

ML-based Neutrino Data Reconstruction Chain
Stage 1-a: input & output
Stage 1-a Input

Stage 1-a Output

37

ML-based Neutrino Data Reconstruction Chain
Stage 1-b: Particle Endpoint Prediction
Point Proposal
Network (PPN)… extension of U-ResNetwith 3 CNN blocks

See arxiv:2006.14745

ML-based Neutrino Data Reconstruction Chain
Stage 1-b: Particle Endpoint Prediction

PPN1 generates anattention mask at thelowest resolution
See arxiv:2006.14745

ML-based Neutrino Data Reconstruction Chain
Stage 1-b: Particle Endpoint Prediction

PPN2 generates anattention mask at theintermediate resolution
See arxiv:2006.14745

ML-based Neutrino Data Reconstruction Chain
Stage 1-b: Particle Endpoint Prediction

PPN makes the final
prediction (point type +coordinate regression)
See arxiv:2006.14745

ML-based Neutrino Data Reconstruction Chain
Stage 1-b: Particle Endpoint Prediction
96.8% of predicted points within 3 voxels of a true point● 68% of true points found within the radius of 0.12 cm● Traditional (nominal) reconstruction method finds 90% of predicted points within
17 voxels, and 68% of true points found within the radius of 0.74cm

See arxiv:2006.14745

ML-based Neutrino Data Reconstruction Chain
Stage 1: input & output
Stage 1 Input

Stage 1 Output

43

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: Dense Pixel Clustering
Simple approach: path-finding between PPN points● MST to find the “shortest” path between PPN points to cluster pixels● Works well! BUT it depends on PPN performance directly + not learnableML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Learnable approach: clustering in the embedding space● Use CNN to learn a transformation function from the 3D voxels to the embedding
space where clustering can be performed in a simple manner

Image credit: arXiv 1708.02551

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: Dense Pixel Clustering
Scalable Particle
Instance Clustering
using Embedding(SPICE)● Embedding decoder learns
transformation● Seediness decoder
identifies the centroids● During training, loss isconditioned so that the
points that belong to thesame cluster follow anormal distribution

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: Dense Pixel Clustering

See arxiv:2007.03083

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: Dense Pixel Clustering

See arxiv:2007.03083

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: Dense Pixel Clustering
Pixels clustered into trajectoryfragments using SPICE

See arxiv:2007.03083

ML-based Neutrino Data Reconstruction Chain
Stage 2-a: input & output
Stage 2-a Input

Stage 2-a Output

50

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Identifying 1 shower ... which consists of many fragments

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Identifying 1 shower ... which consists of many fragments● Interpret each fragment as a graph node + edges connect nodes in the same cluster

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Identifying 1 shower ... which consists of many fragments● Interpret each fragment as a graph node + edges connect nodes in the same cluster● Cast the problem to a classification of node (e.g. particle type) and edge (clustering)ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Graph-NN for Particle
Aggregation (GrapPA)
Input:● Fragmented EM showers

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Graph-NN for Particle
Aggregation (GrapPA)
Input:● Fragmented EM showers

Node features:● Centroid, Covariance matrix, PCA● Start point, direction (PPN)

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Graph-NN for Particle
Aggregation (GrapPA)
Input:● Fragmented EM showers

Node features:● Centroid, Covariance matrix, PCA● Start point, direction (PPN)

Input graph:● Connect every node with every other node(complete graph)

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Graph-NN for Particle
Aggregation (GrapPA)
Input:● Fragmented EM showers

Node features:● Centroid, Covariance matrix, PCA● Start point, direction (PPN)

Input graph:● Connect every node with every other node(complete graph)

Edge features:● Displacement vector● Closest points of approach

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Graph-NN for Particle
Aggregation (GrapPA)
Message passing (MP):● Meta layer (arxiv:1806.01261)● Essentially two 3-layer MLPs (BatchNorm+ LeakyReLU) for edge feature update andnode feature update● 3 times MP (=Edge+Node feature update)

Target:● Prediction of adjacency matrix
representing valid edges (=true partition)● Apply cross-entropy loss

For more studies, see our paper
See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Target Label

Edge Prediction

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Clustering using GrapPA

Edge Prediction● Mean purity and efficiency > 99%● Sufficient for moving to the nextstage (particle analysis)

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2-b: Sparse Fragment Clustering
Start ID using GrapPA

Node prediction● Important to identify the “primary
fragment” (=shower start)● >99% classification accuracy

See arxiv:2007.01335

ML-based Neutrino Data Reconstruction Chain
Stage 2: input & output
Stage 2 Input

Stage 2 Output

62

ML-based Neutrino Data Reconstruction Chain
Stage 3: Interaction Clustering
Identifying Each Interaction?
This task can be casted to the same task
already solved using GrapPA!
● Interaction = a group of particles thatshared the same origin (i.e. neutrino
interaction)● Edge classification to identify aninteraction● Node classification for particle typeID

ML-based Neutrino Data Reconstruction Chain
Stage 3: Interaction Clustering
Target Group

Predicted Interaction

ML-based Neutrino Data Reconstruction Chain
Stage 3: Interaction Clustering
Predicted Interaction

Promising result to address
DUNE-ND reconstruction challenge(~20 neutrino pile-up)

ML-based Neutrino Data Reconstruction Chain
Stage 3: input & output
Stage 3 Input

Stage 3 Output

66

… wrapping up …

67ML-based Neutrino Data Reconstruction Chain
Wrapping up...

End-to-End optimizable full chain!
● ~1 week to train the full chain on a single V100 GPU● Traditional reconstruction chain takes dozens ofpeople, months to years long effort

Machine Learning & Computer Vision in Neutrino Physics
WAKE UP WAKE UP WAKE UP

Summary● Neutrino detector trend: hi-res. particle imaging● Analysis trend: computer vision algorithms○ Benefit the hi-resolution image = lots of heuristics (in non-ML)○ ML-based approach has shown strong promise● ML-based data reconstruction approach○ especially for “busy” detectors … my research :)○ Working on implementing inductive-bias/causality (“physics”)● Other active areas: data/sim domain discrepancy adaptation○ minimize the discrepancy, identify the source, quantify uncertainty

FIN
Machine Learning for Particle Image Analysis

Questions?

70

Backup Slides

SPICE

ML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Instance+Semantic Segmentation● Mask R-CNN … a popular solution, many applications in science/industries○ Object (=instance) detection + instance pixel masking in a bounding box

72ML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Instance+Semantic Segmentation● Mask R-CNN … a popular solution, many applications in science/industries○ Object (=instance) detection + instance pixel masking in a bounding box○ Issue: instance distinction is affected by BB position/size○ Another family: Single-Shot-Detection (SSD) based (not covered here)𝜇𝜇Occlusion issue
The overlap rate ofparticles is very highespecially for our signal(neutrinos) with an event73
vertex.

ML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Instance+Semantic Segmentation● Mask R-CNN … a popular solution, many applications in science/industries○ Object (=instance) detection + instance pixel masking in a bounding box○ Issue: instance distinction is affected by BB position/size○ Another family: Single-Shot-Detection (SSD) based (not covered here)𝜇𝜇Cherry-picked case where
overlap is minimal74

ML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Instance+Semantic Segmentation● Three component loss: pull together points that belong to thesame cluster, keep distance between clusters, and regularization

Equation credit: Dae Hyun K. @ Stanford

Image credit: arXiv 1708.02551

ML-based Neutrino Data Reconstruction Chain
Stage 2: Particle & Interaction Clustering
Instance+Semantic Segmentation● Three component loss: pull together points that belong to thesame cluster, keep distance between clusters, and regularization

Input: 3D pixel energy depositions

Output: 3D pixel clusters(DBScan in hyperspace)76

Backup Slides

Image Classification?

Machine Learning & Computer Vision in Neutrino Physics
Image Classifications: a lot of applications
MicroBooNE
Signal/Background

NOvA Neutrino
Event Topology

LArLIAT
Particle Type Identification

e

γ

NEXT
Signal vs. Backgroundμ78Machine Learning & Computer Vision in Neutrino Physics
Image Classifications: a lot of applications
Especially great for: “a rare event in a quiet detector”● Quiet = can assume “almost always neutrino”○ e.g.) no cosmic-ray background● Rare = “only 1 neutrino”

79

Machine Learning & Computer Vision in Neutrino Physics
Image Classifications: a lot of applications
Especially great for: “a rare event in a quiet detector”● Quiet = can assume “almost always neutrino”○ e.g.) no cosmic-ray background● Rare = “only 1 neutrino”○ the same “image classification architecture” can be applied for…
■ neutrino flavor (topology) classification■ energy regression (image to one FP32 value)■ vertex regression (image to three FP32 value)■ etc. ...

80

Machine Learning & Computer Vision in Neutrino Physics
Image Classifications: a lot of applications
Especially great for: “a rare event in a quiet detector”… but most of LArTPC detectors are not …
● MicroBooNE, ICARUS, SBND, ProtoDUNE … physics in next 5 years○ Busy: typically dozens of cosmic rays in each event● DUNE-ND○ Not rare (busy): a dozen of neutrino interaction pile-up in each event81Machine Learning & Computer Vision in Neutrino Physics
Why Data Reconstruction
Image classification/regression: straight to “flavour & energy”

This is muon neutrino.
Energy is 1 GeV.

82

Machine Learning & Computer Vision in Neutrino Physics
Why Data Reconstruction… but also challenging: a huge single-step of information reduction

This is electron neutrino.
Energy is 1 eV.

… would be nice to know why you thought so ...

83Machine Learning & Computer Vision in Neutrino Physics
Why Data Reconstruction

Image Context Identification

84

Machine Learning & Computer Vision in Neutrino Physics
Why Data Reconstruction

Image Context Correlation/Hierarchy Analysis

85

Machine Learning & Computer Vision in Neutrino Physics
Why Data Reconstruction
Detector noise!
Proton,Proton,and muon!

Interaction
vertex!

So this is likely
2p1𝜇 with one
anomaly cluster86

Backup Slides

Segmentation Data

87

Machine Learning & Computer Vision in Neutrino Physics
Object Detection & Semantic Segmentation

Image Context Identification

88Machine Learning & Computer Vision in Neutrino Physics
Hierarchy and Correlation of Context

Image Context Correlation/Hierarchy Analysis

89

Machine Learning & Computer Vision in Neutrino Physics
Object Detection for Neutrino ID

Neutrino Detection w/ R-CNN(MicroBooNE LArTPC)
JINST 12 P03011 (2017)arXiv:1611.05531

νμMicroBooNE
Simulation + Data Overlay

Task: propose a rectangular box that
contains neutrino interaction(location & size)

90Machine Learning & Computer Vision in Neutrino Physics
Semantic Segmentation for Pixel-level Particle ID
Separate electron/positron energy depositions from other types at raw waveform level.
Helps the downstream clustering algorithms (data/sim comp. @ arxiv:1808.07269)

PRD 99 092001
arXiv:1808.07269

91

Machine Learning & Computer Vision in Neutrino Physics
Semantic Segmentation for Pixel-level Particle ID
Architecture: U-Net + Residual Connections
Encoder

Decoder

input
conv
conv-s2-ﬁnc
tconv-s2-fde
conv-fdec

Number of strided
convolutions, convolution
layers, residual connections,differ in impementations
Image credit: Laura Domine @ Stanford

softmax
Residualconnections
Concatenation

Machine Learning & Computer Vision in Neutrino Physics
Fun Playing with Semantic Segmentation

Machine Learning & Computer Vision in Neutrino Physics
Fun Playing with Semantic Segmentation

Localized features at the
pixel-level are useful toinspect correlation ofdata features &algorithm responses

Machine Learning & Computer Vision in Neutrino Physics
Fun Playing with Semantic Segmentation

Localized features at the
pixel-level are useful toinspect correlation ofdata features &algorithm responses

Backup Slides

Why Neutrino

Machine Learning & Computer Vision in Neutrino Physics
Why neutrinos?
Neutrinos are everywhere!
… which makes them the natural probe to the universe and its history

97

Early Universe

Supernova

Atmosphere

Earth (planets)

AGN

Accelerators

Stars

Reactors

Want to detect & understand more of themFirst, understand how neutrinos travel over spacetime (neutrino oscillations)

Machine Learning & Computer Vision in Neutrino Physics
Neutrino Detectors: What’s Important

Neutrino Oscillation Measurement
Use a neutrino source (flavour X), measure flavour Y at the detector

What’s important?
Three important detector features for oscillation measurementeGood Energy
Resolution

Large Mass(scalable)

Particle ID
Capability

Precise Eν reduce
oscillation uncertainty“More” statistics to measure
rare physics process

Better ν identification
background rejection 98