Machine Learning,
Datascience and Neutrino
Physics at Argonne’s
Leadership Computing Facility
Corey Adams
Data Science Group
Argonne Leadership Computing Facility
corey.adams@anl.gov

1

Argonne Leadership Computing Facility* With content from the Datascience
Team at ALCF, including Venkat
Viswanath, Rick Zamora, Huihuo Zheng

Outline— Argonne Leadership Computing Facility— Datascience (And machine learning) at ALCF
— Scaling Machine Learning for High Performance
Computing— Deep Learning for Neutrino Physics— How to get HPC resources at ALCF – we want you to
use these resources!

2

Argonne Leadership Computing Facility

ALCF

3

Argonne Leadership Computing Facility

Argonne Leadership Computing Facility
The Argonne Leadership Computing Facility provides
world-class computing resources to the scientificcommunity.
• Users pursue scientific challenges• Resources fully dedicated to open science• In-house experts to help maximize results

Theta Intel/Cray
4,392 nodes
281,088 cores
69 TiB MCDRAM
824 TiB DDR4
549 TB SSD
Peak flop rate: 11.69 PF
Mira IBM BG/Q

ALCF offers different pipelines based on your computational readiness. Apply to theallocation program that fits your needs.
https://www.alcf.anl.gov
4

Argonne Leadership Computing Facility

49,152 nodes
786,432 cores
786 TB RAM
Peak flop rate: 10 PF

Argonne’s path to Exascale is critical to our nation’s
scientific leadership

ALCF4

2013

2007
557 TeraFLOPS

5

Argonne Leadership Computing Facility

2017
10 PetaFLOPS

2021
11 PetaFLOPSComputing Resources
Focus on Theta
Theta (Cray XC40)• 11.69 PF Peak performance• 4,392 nodes (281,088 cores)
•

•
•
•
•

2nd Generation Intel® Xeon Phi™ Processor (Knights Landing)

843.264 TB DDR4 and 70.272 TB MCDRAM total memory
128 GB SSD on each node
Cray Aries high speed interconnect in dragonfly topology
10 PB Lustre file system, ~200 GB/s throughput

Also available: Mira, as well as other visualization and analytics clusters, and storage systems

6

Argonne Leadership Computing Facility*

Who Uses ALCF?
Allocation Distributions
10% Director’s Discretionary

Up to 20% ASCR Leadership
Computing Challenge

DOE/SC capability
computingUp to 20% ExascaleComputing Program

7

Argonne Leadership Computing Facility

Leadership-class computing

50% INCITE
205 million
node-hours in
CY2018
Dedicated to OpenScience.

ALCF Data Science Program

8

Argonne Leadership Computing Facility

ALCF Data Science Program (ADSP) Overview• Big Data science problems that require the leadership scale and performance• Span computational, experimental and observational sciences• Focus on data science techniques including but not limited to statistics, machine learning, deep learning,
UQ, image processing, graph analytics, complex and interactive workflows• Two-year proposal period and will be renewed annually. Proposals will target science and software
technology scaling for data science• The program started in 2016 and now in the 3rd year
• Yearly call for proposal.
Next deadline – ~June 2019
• https://www.alcf.anl.gov/alcf-data-science-program

9

Argonne Leadership Computing Facility

ALCF Data Science Program (ADSP)
Targets Data & Learning Pillars

10

Data

Learning• Experimental/observational data
• Image analysis• Multidimensional structure discovery•Complex and interactive workflows•On-demand HPC•Persistent data techniques• Object store• Databases•Streaming/real-time data
•Uncertainty quantification•Statistical methods•Graph analytics• Deep learning• Machine learning steering simulations• Parameter scans• Materials design• Observational signatures• Data-driven models and refinement
for science using ML/DL• Hyperparameter optimization• Pattern recognition• Bridging gaps in theory

Argonne Leadership Computing Facility

Big Data

datascience@alcf.anl.gov

Corey Adams

Prasanna
Balaprakash

Bethany Lusch

Alvaro Vazquez

Taylor Childers

Murali Emani

Elise
Jennings

Xiao-Yong Jin

Murat Keceli

Adrian PopeMisha Salim

Richard Zamora

Huihuo Zheng

Mayagoitia

William Scullin

11

GaneshSivaraman

Argonne Leadership Computing Facility

Tom UramAntonio Villarreal Venkat VishwanathIntegration of Simulation, Data Analytics and Machine
Learning on supercomputers
Traditional
HPC
Systems

Large-Scale
Numerical
Simulation

Scalable
Data
Analytics

Deep
Learning

12

Argonne Leadership Computing Facility

CORAL
Supercomputers
and Exascale
SystemsAurora 2021 - The first US Exascale System

Architecture supports three ways of computing• Large-scale Simulation (PDEs, traditional HPC)
• Data Intensive Applications (scalable science pipelines)• Deep Learning and Emerging Science AI (training and inferencing)
13

Argonne Leadership Computing Facility

Machine Learning and High
Performance Computing

14

Argonne Leadership Computing Facility

Machine Learning and HPC

Accelerate and improve an application’s:
Time to Solution (Training) – with scalable learning techniques, you
can process more images per second, reduce the time per epoch, andreach a trained network faster.
Quality of Solution – with more compute resources available, you canperform hyperparameter searches to optimize network designs andtraining schemes. With powerful accelerators, you can train bigger andmore computationally intense networks.
Inference Throughput – with high bandwidth IO, it is easy to scale
up the throughput of inference techniques for deep learning.

15

Argonne Leadership Computing Facility

High Performance Computing can
improve all aspects of training andinference in machine learning.

Machine Learning and HPC – KNL Nodes
Single Node Performance Matters!
Running a model on an HPC node is often not
like a standard GPU – many configuration
parameters matter, not all models have thesame optimum parameters.
Intel KNL != Nvidia GPU, but KNL can be powerful.
intra_op_parallelism_threads: Nodes that can use
multiple threads to parallelize their execution willschedule the individual pieces into this pool.
inter_op_parallelism_threads: All ready nodes arescheduled in this pool.
config = tf.ConfigProto()
config.intra_op_parallelism_threads = num_intra_threads
config.inter_op_parallelism_threads = num_inter_threads
tf.Session(config=config)
16

Argonne Leadership Computing Facility

https://www.tensorflow.org/guide/
performance/overview

Machine Learning and HPC – KNL Nodes
Affinity can play a large role

KMP_BLOCKTIME=0 is optimal for KNL Nodes
KMP_AFFINITY=granularity=fine,verbose,compact,1,0

17

Argonne Leadership Computing Facility

Intel Affinity Guidelines

Machine Learning and HPC – KNL Nodes
Batch Size can be important
Running a model on an HPC node is often not
like a standard GPU – many configuration
parameters matter, not all models have thesame optimum parameters.
Bigger batch size often yields more images/second
throughput (though not always), but the downside isalways more seconds/global step at a large batch size.

AlexnetResnet50

Inception3

Batch Size Dependence

18

Argonne Leadership Computing Facility

Distributed Learning
Machine learning is a very important workflow for current and future supercomputing systems.
How can you accelerate learning with more computing power?

Image from Uber’s Horovod: https://eng.uber.com/horovod/
19

Argonne Leadership Computing Facility

What is Distributed Learning?
A technique to accelerate training
The backpropagation algorithm is unchanged at it’s heart.
Data Parallel learning – with N nodes, replicate your model on each node.
After the forward and backward computations, average the gradients across all
nodes and use the averaged gradients to update the weights. Conceptually,
this multiplies the minibatch size by N.
Model Parallel Learning – for models that don’t fit on a single node, you candivide a single model across multiple locations. The design of distributing amodel is not trivial, but tools are emerging.
Both (“Mesh” training) – Using n nodes for a single model, and N = k*n nodes
for distributed training, you can achieve accelerated training of extremely large
or expensive models.

20

Argonne Leadership Computing Facility

Data Parallel Learning
Each Model gets uniqueinput data and performs
calculations
independently.

All nodes communicate to average gradients.

Training Process
Model

Gradients

Averaged
Gradients

Gradients

Averaged
Gradients

Gradients

Averaged
Gradients

Training Process
IO/Storage

Model

Training Process
Model

Image from Uber’s Horovod: https://eng.uber.com/horovod/

21

Argonne Leadership Computing Facility

Each Node gets it’s own
copy of the model.

Data Parallel Learning
Scaling Challenges

Each Model gets uniqueinput data and performs
calculations
independently.
IO requires organization
to ensure unique batches.

All nodes communicate to average gradients.
Computation stalls during communication:
keeping the communication to computation ratio
small is important for scaling.

Training Process
Model

Gradients

Averaged
Gradients

Training Process
IO/Storage

IO contention with many
nodes requires parallel
IO solutions

Model

Gradients

Averaged
Gradients

Gradients

Averaged
Gradients

Training Process
Model

Image from Uber’s Horovod: https://eng.uber.com/horovod/

22

Argonne Leadership Computing Facility

Each Node gets it’s own
copy of the model.
Initialization must be
identical or synchronized,
andcheckpointing/summary
information must be
managed with just one
node.

Data Parallel Learning

Image from Uber’s Horovod: https://eng.uber.com/horovod/

23

Argonne Leadership Computing Facility

Data Parallel Learning
Horovod
The simplest technique for data parallel learning
1.

Initialize horovod ( hvd.init() ).

2.

Wrap the optimizer in hvd.DistributedOptimizer.
1.

This uses the underlying optimizer for gradient calculations, and performs an averaging
of all gradients before updating.

2.

Can adjust the learning rate to account for a bigger batch size.

3.

Initialize the networks identically, or broadcast one network’s weights to all others.

4.

Ensure snapshots and summaries are only produced by one rank.

Horovod focuses on handling collective communication so you don’t have to, but let’s you
use all of the tools of your favorite framework. Compatible with mpi4py.

24

Argonne Leadership Computing Facility

Horovod is an open source data
parallel training software compatible
with many common deep learning
frameworks.

Meet Horovod
Github

Horovod Example CodeTensorflow
import tensorflow as tf
import horovod.tensorflow as hvd
layers = tf.contrib.layers
learn = tf.contrib.learn
def main():# Horovod: initialize Horovod.
hvd.init()# Download and load MNIST dataset.
mnist = learn.datasets.mnist.read_data_sets('MNIST-data-%d' % hvd.rank())# Horovod: adjust learning rate based on number of GPUs.
opt = tf.train.RMSPropOptimizer(0.001 * hvd.size())# Horovod: add Horovod Distributed Optimizer
opt = hvd.DistributedOptimizer(opt)hooks = [hvd.BroadcastGlobalVariablesHook(0),tf.train.StopAtStepHook(last_step=20000 // hvd.size()),
tf.train.LoggingTensorHook(tensors={'step': global_step, 'loss': loss},every_n_iter=10),]
checkpoint_dir = './checkpoints' if hvd.rank() == 0 else Nonewith tf.train.MonitoredTrainingSession(checkpoint_dir=checkpoint_dir,hooks=hooks,
config=config) as mon_sess25

Argonne Leadership Computing Facility

Horovod Example CodePytorchimport torch.nn as nn
import horovod.torch as hvd
hvd.init()
train_dataset = datasets.MNIST('data-%d' % hvd.rank(), train=True, download=True,transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307,), (0.3081,))]))train_sampler = torch.utils.data.distributed.DistributedSampler(
train_dataset, num_replicas=hvd.size(), rank=hvd.rank())
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=args.batch_size, sampler=train_sampler, **kwargs)# Horovod: broadcast parameters.
hvd.broadcast_parameters(model.state_dict(), root_rank=0)# Horovod: scale learning rate by the number of GPUs.
optimizer = optim.SGD(model.parameters(), lr=args.lr * hvd.size(),momentum=args.momentum)!
# Horovod: wrap optimizer with DistributedOptimizer.
optimizer = hvd.DistributedOptimizer(optimizer, named_parameters=model.named_parameters())

26

Argonne Leadership Computing Facility

Horovod Example Code
Keras
import keras
import tensorflow as tf
import horovod.keras as hvd# Horovod: initialize Horovod.
hvd.init()# Horovod: adjust learning rate based on number of GPUs.
opt = keras.optimizers.Adadelta(1.0 * hvd.size())# Horovod: add Horovod Distributed Optimizer.
opt = hvd.DistributedOptimizer(opt)model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=opt,
metrics=['accuracy'])
callbacks = [# Horovod: broadcast initial variable states from rank 0 to all other processes.
hvd.callbacks.BroadcastGlobalVariablesCallback(0),]# Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.
if hvd.rank() == 0:
callbacks.append(keras.callbacks.ModelCheckpoint('./checkpoint-{epoch}.h5’))
model.fit(x_train, y_train, batch_size=batch_size,
callbacks=callbacks,
epochs=epochs,
verbose=1, validation_data=(x_test, y_test))
27

Argonne Leadership Computing Facility

Effects of Distributed Learning
1. Increased Batch size means improved estimate of gradients.
1. Scale by N nodes? Sqrt(N)?
2. Scale in a layerwise way? Layerwise Adaptive Rate Scaling (LARS)
2. Increased learning rate can require warm up iterations.
1. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
3. Bigger minibatch means less iterations for the same number of epochs.
1. May need to train for more epochs if another change is not made like boosting thelearning rate.

28

Argonne Leadership Computing FacilityMesh Learning
Tensorflow Mesh
When data-parallel isn’t enough…
Why might you need a Mesh?
• Memory limitations due to network size (number of parameters)• Memory limitations due to input size (massive images, etc)
Mesh Scaling is not trivial:• Computations need to be distributed in an intelligent way to prevent idle nodes• Communication needs to happen frequently during both the forward/backward pass
• Message passing organization details arise from forward/backward small-group
communications and multi-group communications
Expect mesh scaling to get easier over the next few years (or wait for bigger, more
powerful nodes!)29

Argonne Leadership Computing Facility

https://github.com/tensorflow/meshNeutrino Physics @ ALCF

30

Argonne Leadership Computing Facility

A Crash Course in Neutrino
Physics

31

Argonne Leadership Computing Facility

Particle Physics - Building Blocks

Proton = U + U + D
Neutron = U + D + D+ electron + photon= ALL conventional matter"Sewing the fabric of spacetime"
Everything we know about is made from these particles
32

Argonne Leadership Computing Facility

Particle Interactions

Particle Interactions are really just the exchange of
mediating particles.
33

Argonne Leadership Computing Facility

Basic Rules
Fundamental Particles follow basic rules:
1.
2.

Anything not forbidden is allowed.
Things that are forbidden are forbidden by"conservation" laws.
1. Traditional conservation laws: conservation of
momentum, energy, charge, etc...
2. Particle Physics conservation laws: conservation of"lepton" number, "baryon" number, "flavor", "color",...

Neutrinos are one of the biggest rule breakers in the
Universe.
34

Argonne Leadership Computing Facility

Neutrino Oscillations
Neutrinos are super weird!
Over time, an electron
stays as an electron.

An electron neutrino canturn into a muon neutrino
And back!

Measuring the frequency and strength of the "oscillations"tells a LOT about neutrinos.

35

Argonne Leadership Computing Facility

Why Study Neutrinos?
Are neutrinos their own antiparticles?
Do neutrinos explain the matter/antimatter asymmetry in
the Universe?
Are there more than 3 types of neutrinos?
Could new type of neutrinos explain dark matter?
Why is the neutrino mass so much less than other particles?

36

Argonne Leadership Computing Facility

36Liquid Argon Time Projection Chambers
The modern neutrino detector of choice for high energy physics is the Liquid Argon Time
Projection Chamber.

37

Argonne Leadership Computing Facility

Liquid Argon Time Projection Chambers

38

Argonne Leadership Computing Facility

“Projection” Chamber – top down

39

Argonne Leadership Computing Facility

“Projection” Chamber – angle 1

40

Argonne Leadership Computing Facility

“Projection” Chamber – angle 2

41

Argonne Leadership Computing Facility

3 Projections of Same Objects

Liquid argon time projection chambers are the detector-of-choice for high energy physics experiments
1.
2.
3.
4.
5.
6.
7.
8.
9.

Argoneut (Fermilab, Decommissioned)
Lariat (Fermilab Testbed, Running)
MicroBooNE (Fermilab Running)
SBND (Fermilab, Construction)
ICARUS (Fermilab/INFN, Assembly)
ProtoDUNE SP (CERN, Running)
ProtoDUNE DP (CERN, Assemly)
DUNE ND (Fermilab, Design) – pixel readout
DUNE FD (Sanford Lab, SD, Design)

42

Argonne Leadership Computing Facility

Many other physics experiments that needfine-grained tracking and detailed calorimetry use
similar time projection chamber technology.
Novel designs will replace the sense wires withpixel planes for intrinsically 3D imaging.

Promising Technology for applications of Deep
Learning and HPC
Large data size ( O(10^7) pixels per image – for example, 1260 x 2048 pixels, 3 images = 1 Event)Huge datasets (millions of images of both simulated and real data, per experiment)
Multiple fundamental science applications from neutrino oscillations, proton decay searches,beyond-standard-model physics searches
Multiple deep learning techniques directly applicable
Electron vs Photon (Classification)
JINST 12, P03011 (2017).

A Deep Neural Network for Pixel-Level Electromagnetic Particle Identification in
the MicroBooNE Liquid Argon Time Projection Chamber
43

Argonne Leadership Computing Facility

More Data, More Problems
Large scale, high resolution detectors don't fit into any
successful paradigm of analysis.
For the MicroBooNE detector (Fermilab):
Each event is ~40MB of raw data for ~5ms ofdata. 8 GB/s of data (almost 30 TB per hour!)Not feasible!
Need to develop a fast, automated patternrecognition.
CERN

44

Argonne Leadership Computing Facility

More Data, More Problems
Large scale, high resolution detectors don't fit into any
successful paradigm of analysis

UC Irvine Physical Sciences

Complex data requires complex pattern recognition.
45

Argonne Leadership Computing Facility

45

Physicist vs. The Machine

ICPF = Incorrectly Labeled Pixel Fraction.
No classical algorithm achieves this accuracy. Machine Learning out performs
physicist hand labels.

Original

46

Argonne Leadership Computing Facility

Physicist

Network

Cross Plane Information - Classification
How to best utilize multiplane detectors?

Correlations across projections are correlated and learnable.
47

Argonne Leadership Computing Facility

Cosmic Tagging

For detectors on the surface of the earth, theparticles that produce ionization from neutrino
interactions (electrons, protons, muons, photons…) are the same ones that produce ioninzationfrom cosmic rays.

Cosmic muon backgrounds are problematic.
The signatures in the detector are
indistinguishable without pattern recognition, but
cosmic particles are much more common (manyorders of magnitude)

Short Baseline Near Detector (FNAL)

48

Argonne Leadership Computing Facility

Cross Plane Information

Network

49

Argonne Leadership Computing Facility

Cross Plane Information - Segmentation
How to best utilize multiplane detectors?

Correlations across projections are correlated and learnable.
50

Argonne Leadership Computing Facility

Multiplane Segmentation

Original

Network Input

51

Argonne Leadership Computing Facility

Prediction

Truth

Network Output(No cosmics!)

Segmentation: Zoom on the Neutrino

Original

Prediction

Truth

Neutrino passing rate: 80%
Background-only: 6%
Neutrino Purity: 89%
There is no corresponding classical algorithm
that achieves this result.
52

Argonne Leadership Computing Facility

Natively 3D Detectors
Most LArTPCs at large scale are multi projection, 2Ddetectors - but pixelated, 3D detectors are coming.
An obvious question: are the performance gains worth theextra costs?
3mm /12mm /
voxel

voxel

Deep Learning offers a way to quickly
evaluate a configuration and measure it’s
physics reach, in order to measure theperformancegains
of a configuration
vs.
4x better
resolution
is 16xmore
in electronics
andexpected
costs.
DAQ hardware

53

Argonne Leadership Computing Facility

Sparse vs. Dense Convolutions
Deep learning in neutrino datasets could benefit dramaticallyfrom an intelligent application of sparse techniques tomachine learning.
• Save memory by not storing activations on empty pixel
sites• Save computations by ignoring non-active sites.
Neutrino images from liquidargon time projections chambers
are very sparse.
Occupied pixel fraction is:< 0.01 in 2D at 5122 pixels< 5e-5 in 3D at 5123 pixels

54

Argonne Leadership Computing Facility

Sparse Machine Learning Implementations
Details vary, but there is some literature on convolutional neural networks
on spatially sparse data.
1.
2.
3.
4.
5.
6.

55

SBNET (from Uber) - inference only gains in speed by only
feeding forward active blocks of input.
OctNet - sparse convolutions with tree based organization forsparse convolutions
Sparse 3D Conv. Nets - hashmap based convolutions for managing
sparse representations
Submanifold Sparse Convolutional Networks - follow up on 3,
deals with dilation as well.
(PointNet - graph network that works OK on sparse data)(Dynamic Graph CNN - graph network with convolution-like
operation)

Argonne Leadership Computing Facility

Submanifold Sparse Convolutional Networks
This technique brings three new convolution operators that differ fromtraditional convolutions:• Sparse Convolutions operate only on active input sites, butbehave traditionally on the output layer.
• Submanifold Sparse Convolutions maintain the same set of
active sites on the input and output layers (and are “valid” - nochange to spatial size)• Sparse Deconvolutions invert sparse convolutions by reversing
the mapping from input to output of a sparse convolution, usefulin UNet or FCN segmentation architectures.

56

Argonne Leadership Computing Facility

Submanifold Sparse Convolutional Networks
This technique brings three new convolution operators that differ fromtraditional convolutions:• Sparse Convolutions operate only on active input sites, butbehave traditionally on the output layer.
• Submanifold Sparse Convolutions maintain the same set of
active sites on the input and output layers (and are “valid” - nochange to spatial size)• Sparse Deconvolutions invert sparse convolutions by reversing
the mapping from input to output of a sparse convolution, usefulin UNet or FCN segmentation architectures.

57

Argonne Leadership Computing Facility

Sparse Convolutional

58

Argonne Leadership Computing Facility

Submanifold Sparse Convolutional

59

Argonne Leadership Computing Facility

Sparse/Dense Comparison

Roughly 18x speedup in training on identical batch sizes, 58x speedup in peaksingle-GPU throughput.

60

Argonne Leadership Computing Facility

Sparse/Dense Comparison

60 Epochs (40 in 3D) on 64 GPUs in just 4 hours! Approaches state-of-the-art
performance, quantitative study starting to benchmark accuracy of dense vs. sparse
implementations.
61

Argonne Leadership Computing Facility

Sparse Convolutions
Relatively niche computer vision application could be very impactful in particle
physics:• Dramatic speed up of both training and inference times• Substantial reduction of memory impact means less powerful machines can perform
inference• Scalable to CPUs of open science grid
• Neutrino datasets can feed back to the computer vision community for developmentof new techniques (instance detection?) with sparse convolutions.

62

Argonne Leadership Computing Facility

Where is the field going?

63

Argonne Leadership Computing Facility

Full Event Pattern Recognition
Instance aware, cross-plane particle segmentation.

Already, successful demonstrations of segmentation, instanceaware predictions, and multi-plane networks have succeeded.

64

Argonne Leadership Computing Facility

Online Triggers
MicroBooNE data rate is ~8 GB/s. New detectors will only
be bigger and faster - how to keep up with the data stream?

Deep convolutional neural networks offer a
way to dramatically reduce background data
in an online manner, within a DAQ system.

Background Only

Neutrino Interaction

Requires a different data pipeline, but is feasible for
future experiments.
65

Argonne Leadership Computing Facility

Rare Event Searches
In the Deep Underground Physics Experiment, there are
exciting physics studies to do that require powerful analysis
techinques.
J. Hewes Thesis, 2017 (http://inspirehep.net/record/1662648)

Proton Decay

66

Argonne Leadership Computing Facility

neutron toanti-neutron
oscillation

Collaborative Development
Many experiments (and many students!) are interested in deep learning, but applications toneutrino physics are still emerging and common tools unavailable.
Response: deeplearnphysics.org to share open source development:• Open source data format (and open data set)
• OpenGL Interactive Visualization Tools• Open source network implementations• Extensive Tutorials• Slack channel for news, questions, etc.
• Cross-experimental meetings

67

Argonne Leadership Computing Facility

DeepLearnPhysics
When Professor-of-Physics wants their new student to use
deep learning in their thesis, where to turn?
Open Data Set

Model Zoo

• Freely available data
• Open sourcefrom authentic neutrino
implementations ofsimulation
important networks for• Tutorials andphysics
walkthroughs on how to• Evaluation andgenerate
pre-trained weights forapplication-specificopen data
datasets
Goal: Papers released early 2019
68

Argonne Leadership Computing Facility

Open Data Set• Data set is public• Descriptive Paper is in 2nddraft• Model zoo is coming!

69

Argonne Leadership Computing Facility

DUNE
Neutrino Physics is investing into large, liquid argon imaging
detectors.

Deep learning will be THE tool that meets the science needsof DUNE.
70

Argonne Leadership Computing Facility

Your Science @ ALCF

71

Argonne Leadership Computing Facility

Three primary ways for access to LCF
Distribution of allocable hours
10% Director’s Discretionary

Up to 20% ASCR Leadership
Computing Challenge

DOE/SC capability
computingUp to 20% ExascaleComputing Program

72

Argonne Leadership Computing Facility

Leadership-class computing

50% INCITE
205 million
node-hours in
CY2018
Allocations
switching fromcores-hours to
node-hours

Software● ML/DL:● TensorFlow, Keras, Neon, MXNet, PyTorch, Sci-kit
Learn, Graph Analytics (Cray Graph Engine),
Horovod…
● With performance libraries e.g. Intel MKL, MKL-DNN,
LibXSMM etc enabled● Intel optimized Tensorflow● Conda package on Theta● Intel Distribution for Python’s optimized numpy

73

Argonne Leadership Computing Facility

Software● Workflow/Data analysis:● Containers● Singularity container solution for application science workloads● Environment imported into container● mount additional directories into the container with the -B flag
● aprun -n $RANKS -N 1 singularity exec my_image.img ./my_binary● Balsam● Jupyter Hub, MongoDB, Apache Spark, R● Python● Intel and Cray modules on Theta● ALCF alcfpython/2.7.14-20180131● Visualization: Paraview on Theta
74

Argonne Leadership Computing Facility

BALSAM Workflow manager and edge serviceS
for alcf systems
https://www.alcf.anl.gov/balsam

HEP

phase-space
integration job
description

1

messagequeue

Balsam(supervisor)

Balsam(service)

messagequeue

2

HEP-local cluster

event
generation job
description

scheduler(HTCondor)

The Impact

ALCF-Mira• Delivered >150M Core hours for production
science on ALCF systems
3• The ATLAS experiment has used Balsam torun hundreds of millions of compute hoursof event generation jobs on ALCF systems.
Schematic of ATLAS deployment of Balsam on multiple resources to execute a
workflow with alternating serial and parallel stages
ALCF contribution to ATLAS computing ranks
as the 6th largest country in the world.
Balsam is a workflow manager that simplifies the task of• The DIII-D National Fusion Facility usedrunning large-scale job campaigns on ALCF resources while
Balsam to trigger experiment-time analyses
minimizing user involvement and improving productivity.
during Tokamak operation, running more
It interacts closely with job schedulers to optimize job
complex analyses in less time, leading to higher
throughput via individual jobs and ensemble jobs, staging data experiment productivity
in and out as needed. A supervisor component manages• We are investigating use of Balsam for several
execution across multiple compute resources.
other projects, including real-time APS/ALCF,
and in ADSP and ECP projects
messagequeue

Balsam(service)particle
showering job
description

messagequeue

75

Argonne Leadership Computing Facility

scheduler(Cobalt)

HEP-local cluster
Balsam(service)scheduler(HTCondor)

DeepHyper: Scalable hyperparameter search for deep learning
P. Balaprakash, E. Jennings, M. Salim, T. Uram, S. Wild, V. Vishwanath– Development of DL algorithms is time consuming and asignificant portion is spend on tuning and optimizing thehyperparameters such as number of layers number of units
learning rate optimizer epochs,– Model-based search iteratively refines the model in promising
input region by obtaining new outputs at unevaluated input
configurations– Framework:• Initialization phase▪ Random or Latin hypercube sampling• Iterative phase▪ Fit model▪ Sample using the model– Integration with the Balsam workflow enables for improved
used productivity and performance. This facilitates job
submission, data staging, model checkpointing, resource
allocation, etc. Scaled to 1024 Theta nodes and 64 Cooley
nodes
76

Argonne Leadership Computing Facility

Example Surrogate Model Fitted to Sampled Performance(iterative refinement improves the learning model)Map To Allocation Programs
Program
INCITE

Production

Capability Computing

ALCC

Production

SC Capability Computing missions driven

Discretionary

Production

Development, testing, proposal preparation

Data Science Program Production

Developing technical and science capability fordata and learning based workflows

Early Science Program

Next Generation

Developing technical and science capability for
next generation systems

Exascale Computing
Projects

Next Generation

The ECP mission-driven

https://www.alcf.anl.gov/user-guides/how-get-allocation
77

Argonne Leadership Computing Facility

ALCF User Training Workshops and Opportunities
https://www.alcf.anl.g
ov/training

Most Recently: October 2-4, 2018

Argonne Training Program on
Extreme-Scale Computing (ATPESC)
https://extremecomputingtraining.anl.go
v/application/
April 30 - May 2, 2019 Registration
78

Argonne Leadership Computing Facility

Thank You!!!

Feel free to contact us:
corey.adams@anl.gov
datascience@alcf.anl.gov
https://www.alcf.anl.gov/alcf-data-science-program

79

Argonne Leadership Computing Facility

Extra Materials

80

Argonne Leadership Computing Facility

Finding Neutrinos

81

Argonne Leadership Computing Facility

First Steps: Nova"A Convolutional Neural Network Neutrino Event Classifier"(https://arxiv.org/abs/1604.01444)

The Nova experiment trained amodifed version of AlexNet toclassify their neutrino
interactions by type of neutrino.

The convolutional neural network performed as wellor better than traditional pattern recognition in allcases.
The primary physics search saw an efficiency boost of40%.
82

Argonne Leadership Computing Facility

MicroBooNE: Particle ID
SampleElectron

PhotonMuon

Pion

ProtonDetection
Accuracy (%)

77.8 +/- 0.7

83.4 +/- 0.6

89.7 +/- 0.5

71.0 +/- 0.7

91.2 +/- 0.5

Most
frequuentMisID (%)γ (19.9)e- (15.0)

π- (5.4)

μ- (22.6)

μ- (4.6)

Using AlexNet, can individual particles be distinguished? Yes...
JINST 12, P03011 (2017).
Electron

83

Argonne Leadership Computing Facility

PhotonMicroBooNE: Particle ID
SampleElectron

PhotonMuon

Pion

ProtonDetection
Accuracy (%)

77.8 +/- 0.7

83.4 +/- 0.6

89.7 +/- 0.5

71.0 +/- 0.7

91.2 +/- 0.5

Most
frequuentMisID (%)γ (19.9)e- (15.0)

π- (5.4)

μ- (22.6)

μ- (4.6)

Using AlexNet, can individual particles be distinguished? Yes...
JINST 12, P03011 (2017).
Pion

84

Argonne Leadership Computing Facility

ElectronMuon

ProtonMicroBooNE: Particle ID
Electron

Pion

85

Argonne Leadership Computing Facility

Muon

PhotonProtonSemantic Segmentation

A Deep Neural Network for Pixel-Level Electromagnetic Particle Identification in the MicroBooNE Liquid
Argon Time Projection Chamber(https://arxiv.org/pdf/1808.07269.pdf)

86

Argonne Leadership Computing Facility

Semantic Segmentation
In uncrowded
interactions, this isenough to augment
traditional track-fitting
algorithms.

How do you evaluate
performance on data?

87

Argonne Leadership Computing Facility

Data Validation
Even with hand-labeling, how to find a benchmark data set?
Leverage existing physics analyses for network valdiation.
Cosmic Muon Decay Events

Comparison between networklabels and physicist labels.
88

Argonne Leadership Computing Facility

Data Validation

Original

Event 2

Event 1

Even with hand-labeling, how to find a benchmark data set?
Leverage existing physics analyses for network valdiation.

89

Argonne Leadership Computing Facility

Physicist

NetworkPhysicist vs. The Machine

Original

90

Argonne Leadership Computing Facility

Physicist

NetworkPoint Prediction
Network features for
track/shower
identification should
also be useful for
instance aware tasks.
Can a Region Proposal
Network predict thestart and end points ofeach particle?

91

Argonne Leadership Computing Facility

Point Prediction
Network features for
track/shower
identification should
also be useful for
instance aware tasks.
Can a Region Proposal
Network predict thestart and end points ofeach particle?
Yes
L. Domine, K. Terao, SLAC

92

Argonne Leadership Computing Facility

Segmentation in 3D
Meanwhile, extending segmentation networks to 3D has
proven feasible and results are encouraging:

Better resolution gives better results, but GPU memory
rapidly becomes an issue - difficult to train.
93

Argonne Leadership Computing Facility

Convolution Implementation
Given a set of sparse input, in an N dimensional volume, the input data is represented
as m features over a active locations and a coordinates in N dimensional space.
1.

Build a hashmap to convert a spatial coordinate into an array index. Map
each N dimensional active location to a single row index in an a x m matrix.
2. For each spatial location in a filter (3x3 = 9 locations, for example), create alist for each spatial (input, output) pair for that filter location.
1. Submanifold convolutions enforce output locations to be in the input
feature set, sparse convolutions allow all valid output locations
3. For input location i that maps to output location j, perform the matrix
multiply of the (input row i )*(m x n) filter for that spatial location, and add
it to output row j.

94

Argonne Leadership Computing Facility