Machine Learning at LCLS
Locating Detector Features
David Schneider, davidsch@slac.stanford.edu

Detector Images
Sample/Shot Interaction

Shot Characterization

Find Bragg’speaks in a
Diffraction shotat CXIFind Time resolved power during
a shot at AMO

Find delay between pump andprobe during shot at XPP
4

Detector Features: Classification and Regression
Shot characterization requires
detailed Image Processing 1. Base –LCLS provides manydetector features(peak finders, GUI fortuning peaks, Time Tool)

Two color AMO
experimentenergyvalue atwhich highcolor lased2. High color
has lased

Problem: Experiments change.
New image processing is time
consuming to develop
Under study: determining when
Machine learning is more
effective at shot characterization

3. Base low energy
4. Low
Present

5. Femtosecond
difference inarrival times
5Challenges with Machine Learning• Labels for Training data

• LCLS data - no simulator• 100k shots for a typical experiment• In contrast• HEP - standard model• rich set of simulated data for supervised learning• Processing LCLS Data

• Large datasets - megapixel images• Data format – like binary XML•

expensive to load event• detector corrections• Parse all the machine data• For ML, presently, make a copy to hdf5
6

Two Color AMO Experiment (R. Coffee et all)
Replaces Image processing with 1D Peak Finding
Special detector:
high/low energy lasing
early/late peaks on spectrometerenergy level – peak position
Issue - same detector needed forsample/shot interaction
Experiment:
Training runs
Detector used for shot characterization
classification: 4 labels from peaks(neither, low, high, both)
Regression: 2 energy values
Molecular runs
Predict shot (0,1,2,3, low/high energy)
sort down to see structure7M. Mongia, R. Coffee, C.O’ Grady, D. Schneider

Input:
Image + machine data
stable between training
and predictionDense Layers

Concatenate
machine data withflattened CNN
output before final
dense layers

By Aphex34 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374
8

Guided Back Propagation for Feature Finding
Question: why did the Neural
Network make it’s choice? Will
scientists trust these models?
Guided back propagation – intuition
is you ask the network how tochange the image to improve theprediction
Below – technique highlights exactly
what is important
Locate Lasing from G. Backprop?
Still need femtosecond delay
between lasing, not on spectrometer.
Can we robustly get it from guidedbackprop?
9Machine Learning from Reference vs. Signal?
Special detector gave us 4 labels for machine learning
May not generalize to other experiments
However – many experiments record:reference runs to compare to signal runs
General XTCAV Analysis:

Same pattern for Time Tool, Calibrating Detectors, Crystallography

10Reference vs. Signal, Saliency style Algorithms
0/1 Classifier for Two color AMO experiment
1 (high), 2 (low), 3 (both) -> 1 (signal)
but old 1 (high) weaker signal (dataset not balanced)
Guided Backprop: no robust result with it yet
More recent, similar:
Relevance Propagation(A. Binder, et all) 2016
Formulas tricky toimplement for CNN’s
Others: LIME, Grad-CAM
11

Can we generate the right reference from a signal shot?

ReferenceMachineLearning
Splitter
Input from signal
shots (lasing,run with samples, etc.)

Signal(Input – Ref)

May tie into machine learning area of Generative Models
recent work such as Adversarial Networks or Variational Auto Encoders

12Generative Models
Training data:faces

Sample fromZ to generate
New facesZ
Low dimensional
representation

Radford et all, ICLR 2016
Images via CS231n slides
13Generative Models: Vector Math
Training data: faces

samples

Radford et all,ICLR 2016
Images via
CS231n slides

Averages in Z spacemath in Z space

Generate Faces
From these Z’s
14

Lasing Vector Math?

L=Lasing average
In latent Z space

N=No Lasing average
In latent Z space

V=N–L
X=Particular
Lasing shotin latent Zspace

could this remove lasing?

X+ V = magically,
the right no lasingreference?
Numerous issues,
But idea is what applicationsof the representation in Z
space are there?

Variational Auto Encoder (VAE)
I (image)
Encoder: Neural Net

Z(latent space)

Z + eps(variation)

Prior on Z
Decoder: Neural Net
I (image)

Loss: two terms: image recovered: (I – I )^2 -> 0latent space looks like prior: (KL divergence term)

VAE on MNIST (from Keras Blog)
I: 784 flattened pixels, values in [0,1]
Encoder: One Dense Layer
Relu Activation
Z: 2 dimZ - interpolation
Decoder: Dense 2 –> 256, Relu
Dense 256 -> 784, Sigmoid

VAE on Two Color AMO Data – Latent Space
Reduce Images: 5000 pixels
728 x 568 -> 100 x 50
Log/Thresh at 300 ADU
center on horiz ROI

Encoder: two dense layers
5000 -> 256 –> 2Decoder: two dense layers
2 -> 256 -> 5000

Latent Space
Z: 2dim – 6000 lasing (red)1000 no lasing (blue)

VAE – Lasing Reconstruction

OrigReconstruction

Orig ReconstructionVAE – Bad No Lasing Reconstruction

OrigReconstruction

Orig ReconstructionVAE Interpolation

Deep Learning Splitter Idea – train like a GAN
I0 (no lasing)I1 (lasing)

F (Nnet): I -> I
Generates correct reference from signal –removes Lasing (like the GAN generator, but no noise input)
F0

F1

D: (separate Nnet): discriminator:
4 way, I0 , I1 , F0 , F1
Loss Terms:
D 4-way
D 2-way 0=(I0, F0, F1)

1= I1

I0 = F0 (don’t alter no-lasing)I1 = F1 minimize changes to lasing to make it no-lasing
but better, model physical process, conversation law? Etc.

22

Transfer Learning
Reuse award winning Network trained on ImageNet
Millionsofimages

1000
Categories(cat, dog,Bus, ect)Weeks of GPU training

What docodewords
say aboutour Detector
images?

Vgg16 – from http://www.cs.toronto.edu/~frossard/post/vgg16/
23

Transfer VGG16 to XTCAV• Take reference and signal runs of XTCAV• Run Images of each through vgg16 to get codewords• Top rows: reference, bottom: signal

Features that distinguishes images (lasing electrons) are not spatially localized
But in codewords – we see vector components that do24

Possible Application: Look up Reference
XTCAV analysis – given signal (lasing) shot, find the correspondingreference (no lasing) shot (first step to isolate lasing).
Transfer Learning approach: minimize Euclidean distance in codeword space25

First Pass: Doesn’t work wellsubtract reference from signal in Image space, but they don’t line up

26

Can we invert a codeword to get the right reference?
• Start with a reference codeword• close to signal• may not be for existing reference shot
• Optimization: invert through vgg layers• Has statistics of references for that layer• Is ‘close’ to the signal shot output for that layer
Signal shotcodeword
Reference
distribution
with point toinvert
27

Occlusion – first pass – not working well
Train classifier to discriminate reference from signal
Occlude different spots of signal shot, classify each
Want lowest score for signal when feature occluded

Actualfeaturesof signal

regionmostlowering
score

Signal shot before occlusion

Occludingthis actualfeature doeslower score
Heat map: signal score as functionof occluded region28

Occlusion – should it work?
Classifier•trained on reference and signal• Has never seen occluded images before• Why would signal score for occluded ’do the right thing’?
• We will look at t-sne plots to evaluate• t-sne – one of several techniques to embed highdimensional data in a low dimensional space29

T-sne for XTCAV Images

Applied toVGG16
codewords

We expect some Lasing shots not to lase – interesting that they clump together
30

Depends on how good classifier ist-sne plot (unsupervised clustering) of convnet codewords
codewords ofsignal afterfeatures
occluded(should looklike reference)originalreference
codewords

Original signal
codewords

31

Depends on how good classifier isIdeal Signal vs. Reference dividing line – occluded looks like reference
Maybe SVM would give us this32

Depends on how good classifier isPoor division for occlusion (still perfect for signal/reference classifier)33

Label Boxes and Train Regression from Codewords

Label 250 images
plot shows variation in labels
White: feature 1
Green: feature 2

First Regression:
codewords tobox for feature 2

Second:codewords tobox for feature 1

Green: Truth
White: predicted

Green: Truth
White: predicted
34

ImageNet Convolutional Networks can Find Beams onAccelerator Screens
250 labeled boxes
VCC screen –high variation
in beamRegression Results:
Measurement of accuracy: for how many
shots is area of intersection/union > 0.5?
VCC Screen – only 0.09 (left)
YAG Screen – 0.89 (not shown)
Critical step: de-noising preprocessing
from A. Rashed35

“DetectorNet” + “ImageNet”
Transfer learning lesson: the neural Network trained on
millions of ImageNet is effective for Detector Data
How much more effective if also trained on millions ofdetector images, from 1000’s of categories like Diffraction ofMolecule A, B, Beam Diagnostic C, etc?

Images from Google Search & slac.stanford.edu/~weaver

36

Image Captioning for Detector Shots?
If we train a Recurrent Neural Network on detector images
with captions, will the machine be able to write them in thefuture?
Laser/Beam interaction for Time Tool

Great shot ofmimi virus diffraction!

X-ray diffraction patternproduced by a cyanobacteriumat the LCLS

Images from Google Search (LCLS public, DESY) & slac.stanford.edu/~weaver

Strange effect on
a CSPAD37