Shape Learning
Dr. Nina Miolane, Postdoctoral Fellow and Lecturer
Stanford Statistics, Pr. Susan Holmes’ Lab(Previous: Inria, BayLabs/CaptionHealth)Bioimaging at Different Scales• Biology: understand mechanisms of life.

Bioimaging at Different Scales• Biology: understand mechanisms of life.
metersOrgansMacroscopic

10#%

Tissues

Cells

10#&

Organelles
Molecules
10#$

Nanoscopic

Bioimaging at Different Scales• Biology: understand mechanisms of life.
metersOrgans10#%

Magnetic resonance imaging (Nobel Prize 2003)
25cm

Tissues

Cells

Macroscopic

Human brain

10#&

Organelles
Molecules
10#$

Nanoscopic

Bioimaging at Different Scales• Biology: understand mechanisms of life.
metersOrgansMacroscopic

10#%

Magnetic resonance imaging (Nobel Prize 2003)
25cm

Tissues

Cells

X-ray computed tomography
Positron emission tomography
Photoacoustic microscopy
Two-photon microscopy
Optical coherence tomography
Confocal microscopy
Etc.

Human brain

10#&

Organelles
40nm

Cryo-electron microscopy (Nobel Prize 2017)

Molecules
10#$

Human ribosome

Nanoscopic

Bioimaging: Shapes Matter• Biology: understand mechanisms of life.
metersOrgansMacroscopic
Shape is linked to function.

10#%
25cm

Tissues

Cells

Human brain

10#&

Organelles
40nm

Molecules
10#$

Human ribosome

Nanoscopic

Bioimaging: Shapes Matter• Biology: understand mechanisms of life.
metersOrgansMacroscopic
Shape is linked to function.

10#%
25cm

Tissues

Cells

Human brain

10

Alzheimer’s#&

Organelles
40nm

Molecules
10#$

Human ribosome

Nanoscopic

HealthyBioimaging: Shapes Matter• Biology: understand mechanisms of life.
metersOrgansMacroscopic
Shape is linked to function.

10#%
25cm

Tissues

Cells

Human brain

10#&

Organelles
40nm

Molecules
10#$

Human ribosome

Nanoscopic

Alzheimer’s

HealthyMisfolded protein

HealthyBioimaging: Shapes Matter• Biology: understand mechanisms of life.
metersOrgansMacroscopic
Shape is linked to function.

10#%
25cm

Tissues

Cells

Human brain

10#&

Organelles
40nm

Molecules
10#$

Human ribosome

Nanoscopic

Alzheimer’s

HealthyMisfolded protein

Healthyà Shape analysis of bioimaging data.

Shape Learning in (Biomedical) Imaging
Advance Biomedical Research with Shape Learning
Automatic diagnosis

Shape modeling

Surgical planning

Shape statistics

Analyze Images and Reconstruct Shapes
Processing

Shape reconstruction

In this talk
Advance Biomedical Research with Shape Learning
Automatic diagnosis

Shape modeling

Surgical planning

Shape statistics

Analyze Images and Reconstruct Shapes
Processing

Shape reconstruction

Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

Goal: Surgical planning of pelvis fracture reduction• Collaboration with orthopedic surgeon in Nice Hospital (France)

State of the art: External fixation

Proposed method: Internal fixation [1]• Medical question [1]:• Save time by analyzing geometry of pelvis & internal fixations (bars, screws)?
• Statistical question [2-4]:• Uncertainty quantification of shape learning with “Procrustes analysis”?
[1] Darmanté, Bugnas, Bernard De Dompsure, Barresi, Miolane, Pennec, de Peretti, Bronsard. Biometric 3D analysis of the pelvis. (2014)[2] Miolane, Pennec. Biased estimators on quotient spaces. International conference on Geometric Science of Information. (2015)[3] Miolane, Holmes, Pennec. Template shape estimation: correcting an asymptotic bias. SIAM Journal of Imaging Science. (2017)[4] Miolane, Holmes, Pennec. Topologically constrained template controls its statistical inconsistency. SIAM Journal of Geometry and Algebra. (2018)

Data and Shape ModelingData:• 100 patients, CT scans, DICOM• 52 male, 48 female, diversity of ages
Shape modeling:• Manual selection of 16 “landmarks” in 3D à Data point 𝐱 ∈ ℝ𝟑 × 𝟏𝟔
• First model of shape: characteristics unchanged by rotations and translations.

Anterior

Posterior

Method: Procrustes Analysis• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).

Statisticallearning
Translate(Scale)[1] Dryden, Mardia, Statistical Shape Analysis, Wiley, Chichester, (1998).

RotateResults: Procrustes Analysis
1. Superimpose
2. Compute mean

Results: Procrustes Analysis
1. Superimpose
2. Compute mean

PC 1
(Size)

PC 2(Width)3. PCA
79% of variance in 6 PCs

Results: Hypothesis Testing
Do we need two sets of fixations? (Male/Female) [1]T-test. 𝑯𝟎 : M, F distributions have equal means.
Kolmogorov-Smirnov test. 𝑯𝟎 : M, F follow the same continuous distribution.

T-test & KS-test
do not reject 𝐻7 :𝑝 = 0.86
and 𝑝 = 0.80

à One set of anterior bar

T-test & KS-test
reject 𝐻7 :𝑝 = 8.8𝑒 − 5and 𝑝 = 0.001

àTwo sets of posterior bars

[1] Darmanté, Bugnas, Bernard De Dompsure, Barresi, Miolane, Pennec, de Peretti, Bronsard. Biometric 3D analysis of the pelvis. (2014)

Research Contributions on Shape Learning• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).

Statisticallearning
Translate(Scale)[1] Dryden, Mardia, Statistical Shape Analysis, Wiley, Chichester, (1998).

RotateResearch Contributions on Shape Learning• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).
Effect of superimposition (preprocessing)?

Effect of scaling?

Research Contributions on Shape Learning• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).
Effect of superimposition (preprocessing)?
à Analysis of mean shape:?
Kendall shapes 1989: No bias.
Le. 1993. No bias on mean « shape » for 2D.
Lele. 1998. Bias on mean « form » for 2D.
Dryden, Kume, Paine, Wood. 2017. Bias for 3D.

Effect of scaling?

Research Contributions on Shape Learning• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).
Effect of superimposition (preprocessing)?

Effect of scaling?

à Analysis of mean shape:?
Kendall shapes 1989: No bias.
Le. 1993. No bias on mean « shape » for 2D.
Lele. 1998. Bias on mean « form » for 2D.
Dryden, Kume, Paine, Wood. 2017. Bias for 3D.

?
Allassonnière, Amit, Trouvé. 2007. Bias on 1D signals.
Kurtek, Srivastava, Wu. 2011. No bias on 1D signals.
Bigot, Charlier. 2011. Bias on 2D curves.

Research Contributions on Shape Learning• Shape Learning with Procrustes analysis [1]:1. Optimally "superimpose” objects: translate, rotate and possibly scale.
2. Perform statistical learning, e.g. Principal Component Analysis (PCA).
Effect of superimposition (preprocessing)?

Effect of scaling?

à Analysis of mean shape:?
Kendall shapes 1989: No bias.
Le. 1993. No bias on mean « shape » for 2D.
Lele. 1998. Bias on mean « form » for 2D.
Dryden, Kume, Paine, Wood. 2017. Bias for 3D.

?
Allassonnière, Amit, Trouvé. 2007. Bias on 1D signals.
Kurtek, Srivastava, Wu. 2011. No bias on 1D signals.
Bigot, Charlier. 2011. Bias on 2D curves.

Why a bias? When? How important? Can we correct it, if needed?

Research Contributions on Shape Learning• Preprocessing biases the analysis [2]• Asymptotic bias [3]GC 𝜃 = − F 𝐻 𝜃 + 𝒪 𝜎L + 𝜖 𝜎
Theorem [Miolane, Holmes, Pennec (2017)]: Bias 𝜃,Hwith 𝐻 𝜃 mean curvature vector of template’s orbit; 𝜖 exponentially decreasing for 𝜎 → 0.
Assumptions: Shape space = finite dim. Riemannian quotiented by isometric action of finite dim. Lie group.

[2] Miolane, Pennec. Biased estimators on quotient spaces. International conference on Geometric Science of Information. (2015)[3] Miolane, Holmes, Pennec. Template shape estimation: correcting an asymptotic bias. SIAM Journal of Imaging Science. (2017)[4] Miolane, Holmes, Pennec. Topologically constrained template controls its statistical inconsistency. SIAM Journal of Geometry and Algebra. (2018)

Research Contributions on Shape Learning• Preprocessing biases the analysis [2]• Asymptotic bias [3]GC 𝜃 = − F 𝐻 𝜃 + 𝒪 𝜎L + 𝜖 𝜎
Theorem [Miolane, Holmes, Pennec (2017)]: Bias 𝜃,Hwith 𝐻 𝜃 mean curvature vector of template’s orbit; 𝜖 exponentially decreasing for 𝜎 → 0.
Assumptions: Shape space = finite dim. Riemannian quotiented by isometric action of finite dim. Lie group.

?
Kendall shapes 1989: No bias.
Le. 1993. No bias on mean « shape » for 2D.
Lele. 1998. Bias on mean « form » for 2D.
Dryden, Kume, Paine, Wood. 2017. Bias for 3D.

?

Allassonnière, Amit, Trouvé. 2007. Bias on 1D signals.
Kurtek, Srivastava, Wu. 2011. No bias on 1D signals.
Bigot, Charlier. 2011. Bias on 2D curves.

[2] Miolane, Pennec. Biased estimators on quotient spaces. International conference on Geometric Science of Information. (2015)[3] Miolane, Holmes, Pennec. Template shape estimation: correcting an asymptotic bias. SIAM Journal of Imaging Science. (2017)[4] Miolane, Holmes, Pennec. Topologically constrained template controls its statistical inconsistency. SIAM Journal of Geometry and Algebra. (2018)

Research Contributions on Shape Learning• Preprocessing biases the analysis [2]• Asymptotic bias [3]GC 𝜃 = − F 𝐻 𝜃 + 𝒪 𝜎L + 𝜖 𝜎
Theorem [Miolane, Holmes, Pennec (2017)]: Bias 𝜃,Hwith 𝐻 𝜃 mean curvature vector of template’s orbit; 𝜖 exponentially decreasing for 𝜎 → 0.
Assumptions: Shape space = finite dim. Riemannian quotiented by isometric action of finite dim. Lie group.

?
Kendall shapes 1989: No bias.
Le. 1993. No bias on mean « shape » for 2D.
Lele. 1998. Bias on mean « form » for 2D.
Dryden, Kume, Paine, Wood. 2017. Bias for 3D.

• Correction methods [3-4]Simulations:True, Estimation, Corrections?

Allassonnière, Amit, Trouvé. 2007. Bias on 1D signals.
Kurtek, Srivastava, Wu. 2011. No bias on 1D signals.
Bigot, Charlier. 2011. Bias on 2D curves.
More bias
Spatial bias
Less bias[2] Miolane, Pennec. Biased estimators on quotient spaces. International conference on Geometric Science of Information. (2015)[3] Miolane, Holmes, Pennec. Template shape estimation: correcting an asymptotic bias. SIAM Journal of Imaging Science. (2017)[4] Miolane, Holmes, Pennec. Topologically constrained template controls its statistical inconsistency. SIAM Journal of Geometry and Algebra. (2018)

Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

Goal: Find Presymptomatic Biomarkers of Alzheimer’s• Alzheimer’s: accelerated aging of the brain (atrophy)

year
ExtrapolatedObserved

Extrapolated• Medical question [1]:• Early diagnosis of Alzheimer’s disease through 𝐴𝛽LH +/- markers?
• Statistical question [2-4]:• How do we perform learning on deformations?
[1] Lorenzi, Ayache, Frisoni, Pennec. Mapping the effects of 𝐴𝛽%#LH levels on the longitudinal changes in healthy aging. (2011)[2] Miolane, Pennec. Statistics on Lie groups: a need to go beyond the pseudo-Riemannian framework. International MaxEnt conference (2014)[3] Miolane, Pennec. Computing bi-invariant pseudo-metrics on Lie groups for consistent statistics. Entropy (2015)[4] Miolane, Holmes, Pennec. Learning weighted Riemannian submanifolds with variational autoencoders. Submitted to CVPR conference (2019)Data and Shape ModelingData:• 98 patients, structural MRIs (ADNI database)• Two groups: 41 𝐴𝛽%#LH + and 57 𝐴𝛽%#LH Shape modeling:• 3D MR image à Data point x ∈ 0, 1 HR&×HR&×%$H• Second model of shape: deformation 𝜙 from a template (cf [3-4]!)

𝜙𝜙𝜙[3] Miolane, Holmes, Pennec. Template shape estimation: correcting an asymptotic bias. SIAM Journal of Imaging Science. (2017)[4] Miolane, Holmes, Pennec. Topologically constrained template controls its statistical inconsistency. SIAM Journal of Geometry and Algebra. (2018)

Method: Smooth Deformations Analysis
Shape Learning with Diffeomorphisms [1-3]:
1. Compute:• Subjects’ atrophy (intrasubject optimal deformation)• Mean atrophy 𝜇7• Morphological difference: each subject <-> mean (intersubject optimal deformation)
2. Statistical learning of deformations, i.e. dataset of deformation fields

Deformation field of
subject 1, aligned on
the mean anatomy[1] Trouvé. Diffeomorphisms Groups and Pattern Matching in Image Analysis. International Journal of Computer Vision. (1998)[2] Vercauteren, Pennec, Perchant, Ayache. Symmetric log-domain diffeomorphic registration: A demons-based approach. MICCAI. (2008)[3] Yang, Kwitt, Styner, Niethammer. Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach. Neuroimage. (2017)

DatasetResults: Analysis & Hypothesis Testing•Statistics on local volume changes [1]:Average deformation fields v

Ventricles:𝑑𝑒𝑡 ∇v = 3.84% for 𝐴𝛽LH –𝑑𝑒𝑡 ∇v = 6.73% for 𝐴𝛽LH +Hippocampus:
𝑑𝑒𝑡 ∇v = 0.14% for 𝐴𝛽LH –𝑑𝑒𝑡 ∇v = 0.24% for 𝐴𝛽LH +[1] Lorenzi, Ayache, Frisoni, Pennec. Mapping the effects of 𝐴𝛽%#LH levels on the longitudinal changes in healthy aging. (2011)

Results: Analysis & Hypothesis Testing•Statistics on local volume changes [1]:Average deformation fields v

Ventricles:𝑑𝑒𝑡 ∇v = 3.84% for 𝐴𝛽LH –𝑑𝑒𝑡 ∇v = 6.73% for 𝐴𝛽LH +Hippocampus:
𝑑𝑒𝑡 ∇v = 0.14% for 𝐴𝛽LH –𝑑𝑒𝑡 ∇v = 0.24% for 𝐴𝛽LH +•𝐴𝛽LH +/- linked to accelerated atrophy? [1]Hotelling’s two samples 𝑻𝟐
𝐻7 : Mean deformations of 𝐴𝛽LH + / - are equal
à Reject 𝐻7 on hippocampus and ventricles.
[1] Lorenzi, Ayache, Frisoni, Pennec. Mapping the effects of 𝐴𝛽%#LH levels on the longitudinal changes in healthy aging. (2011)𝑝 −valueResearch Contributions on Shape Learning
Shape Learning with Diffeomorphisms [1-3]:
à Statistical learning, dataset of deformations

Dataset[1] Trouvé. Diffeomorphisms Groups and Pattern Matching in Image Analysis. International Journal of Computer Vision. (1998)[2] Vercauteren, Pennec, Perchant, Ayache. Symmetric log-domain diffeomorphic registration: A demons-based approach. MICCAI. (2008)[3] Yang, Kwitt, Styner, Niethammer. Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach. Neuroimage. (2017)

Research Contributions on Shape Learning
Shape Learning with Diffeomorphisms [1-3]:
à Statistical learning, dataset of deformations

Dataset• What is the right framework to perform learning on Lie groups?
𝜙_` 𝑠à Analysis of the mean:• [1]: Right-invariant metric mean
• [2]: Bi-invariant exponential barycenter• [4]: Left-invariant metric mean
à Universal definition of mean?

𝜓
𝜙^

𝜓

𝜓 ∘ 𝜙 = 𝜓 ∘ 𝜙^𝜙 ∘ 𝜓 = 𝜙^ ∘ 𝜓[1] Trouvé. Diffeomorphisms Groups and Pattern Matching in Image Analysis. International Journal of Computer Vision. (1998)[2] Vercauteren, Pennec, Perchant, Ayache. Symmetric log-domain diffeomorphic registration: A demons-based approach. MICCAI. (2008)[3] Yang, Kwitt, Styner, Niethammer. Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach. Neuroimage. (2017)[4] Boisvert, Pennec, Labelle, Cheriet, Ayache. Principal spine shape deformation modes using Riemannian geometry. (2006)

Research Contributions on Shape Learning• No universal definition of mean with a metric structure [2-3]Algorithm computing a biinvariant pseudo-metric on a
Lie group, if it exists [3].

[2] Miolane, Pennec. Statistics on Lie groups: a need to go beyond the pseudo-Riemannian framework. International MaxEnt conference (2014)[3] Miolane, Pennec. Computing bi-invariant pseudo-metrics on Lie groups for consistent statistics. Entropy (2015)[4] Miolane, Holmes, Pennec. Learning weighted Riemannian submanifolds with variational autoencoders. Submitted to CVPR conference (2019)

Research Contributions on Shape Learning• No universal definition of mean with a metric structure [2-3]Algorithm computing a biinvariant pseudo-metric on a
Lie group, if it exists [3].

• Use of families of metrics for unsupervised learning [4]
(a)
(b)
(c)
(d)

Principal component analysis,
Principal geodesic analysis,
Nonlinear subspace learning,Nongeodesic subspace learning [4].

[2] Miolane, Pennec. Statistics on Lie groups: a need to go beyond the pseudo-Riemannian framework. International MaxEnt conference (2014)[3] Miolane, Pennec. Computing bi-invariant pseudo-metrics on Lie groups for consistent statistics. Entropy (2015)[4] Miolane, Holmes, Pennec. Learning weighted Riemannian submanifolds with variational autoencoders. Submitted to CVPR conference (2019)

Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

Goal: Prenatal Detection of Neurodevelopmental disorders
Ultrasound (screening)

MRI (detection)• Fetal brain imaging can detect disorders• Fetal MRI slices not aligned•

Questions:• Reconstruction: 3D reconstruction of brain by slice alignment? [1]• Prediction: given a 2D slice, prediction of orientation and position? [1-2][1] Hou, Miolane, et al. Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry. MICCAI (2018).
[2] Caption Health (including Miolane). FDA authorization for Caption Guidance – the world’s first AI-guided medical imaging acquisition system (2020).

Data - Supervised Learning
Dataset from clinically-validated reconstructed 3D volumes:• Train: 29 volumes à 362,000 slices• Test: 5 volumes à 62,500 slices• Ground-truth labels: pose 𝑝 associated to each slice

𝑋_ =; 𝑌_ = 𝑝_ = (𝑅_ , 𝑡_ )𝑖 = 1…𝑛Method: Supervised Pose Prediction with CNNe.g. GoogLeNet

Choice of a loss function?
= Choice of a distance on poses

Method: Choice of loss function• 6D nonlinear space of poses: 𝑆𝐸 3 = 𝑆𝑂 3 × ℝn

Ground truth

Prediction

×Ground truth

Prediction

Method: Choice of loss function• 6D nonlinear space of poses: 𝑆𝐸 3 = 𝑆𝑂 3 × ℝn

Ground truth

Prediction

×Ground truth

Prediction

Method: Choice of loss function• 6D nonlinear space of poses: 𝑆𝐸 3 = 𝑆𝑂 3 × ℝn
Identity

Ground truth

Prediction

×Ground truth
Identity
Prediction• Loss function 𝑑H :• Respect 𝑆𝐸 3 geometry: left-invariant distance: 𝑑 𝑝_ , 𝑝o_ = 𝑑(𝑝7 ∗ 𝑝_ , 𝑝q ∗ 𝑝o_ )• Define an inner-product matrix 𝑍 at the identity• Learn 𝑍 from the data: inverse of errors’ covarianceResults: Comparison with State-of-the-art• State-of-the-art: PoseNet [1] and Anchor Points [2] (linear non-adaptive losses)[1] Kendall, et al.: Posenet: A convolutional network for real-time 6-DOF camera relocalization. ICCV (2015)[2] Hou, et al.: Predicting slice-to-volume transformation in presence of arbitrary subject motion. MICCAI (2017)[3] Hou, Miolane, et al. Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry. MICCAI (2018)

Results: Comparison with State-of-the-art• State-of-the-art: PoseNet [1] and Anchor Points [2] (linear non-adaptive losses)• Pose prediction: no statistical difference (t-test) [3]
Mean error on each pose’s coordinate:𝑝_ ↔ 𝑝o_• 3D brain reconstruction: statistical improvement (t-test) [3]
Accuracy of reconstruction in image metrics:
Image metrics:↔

[1] Kendall, et al.: Posenet: A convolutional network for real-time 6-DOF camera relocalization. ICCV (2015)[2] Hou, et al.: Predicting slice-to-volume transformation in presence of arbitrary subject motion. MICCAI (2017)[3] Hou, Miolane, et al. Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry. MICCAI (2018)

Results: Comparison with State-of-the-art• State-of-the-art: PoseNet [1] and Anchor Points [2] (linear non-adaptive losses)• Pose prediction: no statistical difference (t-test) [3]
Mean error on each pose’s coordinate:𝑝_ ↔ 𝑝o_• 3D brain reconstruction: statistical improvement (t-test) [3]
Accuracy of reconstruction in image metrics:
Image metrics:↔
à Prediction only as accurate as the information in the image
à Interesting performance metric = volume reconstruction[1] Kendall, et al.: Posenet: A convolutional network for real-time 6-DOF camera relocalization. ICCV (2015)[2] Hou, et al.: Predicting slice-to-volume transformation in presence of arbitrary subject motion. MICCAI (2017)[3] Hou, Miolane, et al. Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry. MICCAI (2018)

Research Contributions

Back-propagation of loss: Perform Riemannian geometric computations on GPUs?

[Miolane et al. Geomstats: a python package for Riemannian geometry in machine learning. 2018-2020]

Research Contributions

Back-propagation of loss: Perform Riemannian geometric computations on GPUs?
• Geomstats: Python package for geometry in machine learning [1]• NumPy, TensorFlow, PyTorch backends• Continuous integration (unit testing)• Scikit-learn API

github.com/geomstats
2018[1] Miolane et al. Geomstats: a python package for Riemannian geometry in machine learning. Submitted to SciPy and JMLR. (2018 - present)2020Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

Goal: Biomolecular Reconstruction for Drug Discovery• Stages of drug discovery:• Biomolecular imaging with cryo-electron microscopy•

Questions:• Reconstruction: 3D reconstruction of molecule? [1]• Prediction: given a 2D image, estimation of orientation and position? [1][1] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Data – Unsupervised Learning

𝑋_ =No ground-truth labels here!

; 𝑌_ = 𝑝_ = (𝑅_ , 𝑡_ )𝑖 = 1…𝑛• Data generative model: 𝑥_ = 𝑃𝑆𝐹_ ∗ 𝑃 𝑅_ ⋅ 𝑉 + 𝜖_ , where:• V biomolecular volume,• 𝑅_ rotation,• 𝑃 projection operator,• 𝑃𝑆𝐹_ point-spread function (camera),• 𝜖_ noise.

Simulated data without noise:

Real data with noise:Method: Expectation-Maximization?
Maximum-Likelihood / Maximum a Posteriori in a latent variable model:

𝑥_ = 𝑃𝑆𝐹_ ∗ 𝑃 𝑅_ ⋅ 𝑉 + 𝜖_ ,• Initialization: 𝑉 7• At iteration 𝑘:𝑖 = 1…𝑛Log-likelihood

1. E-step: For each 𝑖 = 1 … 𝑛:𝑞 ∗ { 𝑅_ , 𝑃𝑆𝐹_ = 𝑝 𝑅_ , 𝑃𝑆𝐹_ 𝑥_ , 𝑉C ({) )2. M-step:
𝑉C {|% = argmax Lower Bound(𝑉, 𝑞 ∗ )𝑉until convergence.

C 7

𝑉

C %

𝑉

C H

𝑉

𝑉

Method: Expectation-Maximization?
Maximum-Likelihood / Maximum a Posteriori in a latent variable model:

𝑥_ = 𝑃𝑆𝐹_ ∗ 𝑃 𝑅_ ⋅ 𝑉 + 𝜖_ ,• Initialization: 𝑉• At iteration 𝑘:

Problem:
No closed-form posterior7

1. E-step: For each 𝑖 = 1 … 𝑛:𝑞 ∗ { 𝑅_ , 𝑃𝑆𝐹_ = 𝑝 𝑅_ , 𝑃𝑆𝐹_ 𝑥_ , 𝑉C ({) )2. M-step:
𝑉C {|% = argmax Lower Bound(𝑉, 𝑞 ∗ )𝑉until convergence.

𝑖 = 1…𝑛Solutions:• Sampling [1]or:• Variational inference [2](on average 10x faster)𝑞𝑞q‡ˆ

[1] Scheres. RELION: Implementation of a Bayesian approach to cryo-EM structure determination. Journal of Structural Biology (2012).
[2] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

𝑝(… |𝑥_ )Method: Variational Autoencoders (VAEs)• Training with loss: reconstruction + regularization• Hyperparameter search:• Bayesian optimization [2]• Asynchronous hyperband (scheduler, early stopping) [3]

Encoder: 𝑞‰ 𝑧 𝑥

Decoder: 𝑝‹ 𝑥|𝑧[1] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).
[2] Bergstra Yamins, Cox. Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms (2013).
[3] Li et al. Massively Parallel Hyperparameter Tuning (2018).

Method: Variational Autoencoders (VAEs)• Training with loss: reconstruction + regularization• Hyperparameter search:• Bayesian optimization [2]• Asynchronous hyperband (scheduler, early stopping) [3]

Encoder: 𝑞‰ 𝑧 𝑥

Decoder: 𝑝‹ 𝑥|𝑧

Problem:(𝑅, 𝑃𝑆𝐹) entangled into 𝑧

Disentanglement:• Modified losses: 𝛽-VAE, InfoGAN, 𝛽-TCVAE, dHSIC, VFAE
• Structure in latent space or decoder: FactorVAE, VITAE, spatial-VAE
à Exploit geometric structure of the data [1][1] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).
[2] Bergstra Yamins, Cox. Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms (2013).
[3] Li et al. Massively Parallel Hyperparameter Tuning (2018).

Method: Geometric Interpretation of VAEs
• VAE: Manifold learning algorithm• Proposition [1]:Manifold of noiseless cryo-EM images = SO(2) / isotropyà Disentanglement:• Rotation 𝑅 à polar angle• Defocus 𝑃𝑆𝐹 à radius

Action of 2D rotations on the space of images,
schematically represented as R2.

[1] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Results on Simulated Cryo-EM Images••••

Human ribosome 80s
n = 2,544 cryo-EM images 128x128, no noise
Inplane rotations
Camera parameters: 6 defocuses

Data

Results on Simulated Cryo-EM Images••••

Human ribosome 80s
n = 2,544 cryo-EM images 128x128, no noise
Inplane rotations
Camera parameters: 6 defocuses

Data

Manifold learning ☑

Epoch 0Reconstruction

Epoch 5Epoch 20

Epoch 75

Estimation(rMSE: 1.72 degrees)(rMSE: 0.18𝜇𝑚)

Results on Experimental Cryo-EM Images••••

Human ribosome 80s
n = 5,119 cryo-EM images 128x128
Inplane rotations
Camera parameters not restricted

Data• Miolane et al. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Results on Experimental Cryo-EM Images••••

Human ribosome 80s
n = 5,119 cryo-EM images 128x128
Inplane rotations
Camera parameters not restricted

Data

Manifold learning ☑
Epoch 0Reconstruction

Epoch 5Epoch 20

Epoch 85

• Miolane et al. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Latent Space: Outlier detection

Area under the curve AUC = 0.86

In plane rotation(rMSE: 18.45 degrees)

DefocusRadius in latent space

Polar angle in latent space

Estimation(rMSE: 0.61 𝜇𝑚)

Estimation in reconstruction
Templatefrom sampling

Templatefrom variational inferenceState-of-the-art: RELION [1]Variational Autoencoders [2]• Same reconstruction accuracy (visually), on downsampled dataset• However, variational inference methods on average 10x fasterà Allow to tackle the explosion of data generated by cryo-EM!

[1] Scheres. RELION: Implementation of a Bayesian approach to cryo-EM structure determination. Journal of Structural Biology (2012).
[2] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Outline
I. Advance Biomedical Research with Shape Learning

Surgical planning & Procrustes analysis

Automatic diagnosis & Learning transformations

II. Analyze Images and Reconstruct Shapes
Pose prediction for fetal brain reconstruction
Orientation estimation for cryoEM biomolecules reconstruction

ML at SLACML at SLAC: Particle Physics• Physics: understand elementary particle interactions.
𝜈• + 𝑛 → 𝜇 + 𝑝𝜈• + 𝑛/𝑝 → 𝜇# + 𝑛/𝑝 + mesons

Shape is linked to particle interactions.
Neutrino interactions. Proton Module, T2K (2013).

•
•
•
•

Number of tracks
Color/width
LengthCurvatureà Shape reconstruction & analysis
𝜇# and 𝜋 | candidate event. ArgoNeuT, FermiLab (2019).

ML at SLAC: Instrument Calibration• Hyperparameter search:• Bayesian optimization [1]• Asynchronous hyperband (scheduler, early stopping) [2][1] Bergstra Yamins, Cox. Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms (2013).
[2] Li et al. Massively Parallel Hyperparameter Tuning (2018).

ML at SLAC: Fast Cryo-EM Pipelines
Templatefrom sampling

Templatefrom variational inferenceState-of-the-art: RELION [1]Variational Autoencoders [2]• Generalize method to 3D rotations• Optimize for speed• Benchmark against RELION[1] Scheres. RELION: Implementation of a Bayesian approach to cryo-EM structure determination. Journal of Structural Biology (2012).
[2] Miolane, Poitevin, Li, Holmes. Estimation of Orientation from Cryo-Electron Microscopy with Variational Autoencoders (2019).

Collaborators

Xavier Pennec

Susan Holmes

Frédéric Poitevin

Johan Mathe…and: Jérémie Lequeux, Michael
Jorda, Alice Lebrigant, Nicolas
Guigui, Bernhard Kainz, Nicolas
Sébastien and Benjamin Hou.

Claire Donnat

Bishesh Khanal

Yee-Ting Li.. and colleagues: Laura Symul,
Anne-Maud Ferreira, Pratheepa
Jeganathan, TJ Lane.

Funding:• National Inria Cordi-S Fellowship + International France-Stanford Fellowship• Inria@SiliconValley Postdoc Fellowship + L’Oréal-Unesco Women in Science• National Science Foundation DMS RTG grant (Pr. Holmes’)Shape Learning

Thank you for your attention