A Survey of Visualization and Analysis inHigh-Resolution Connectomics
Johanna Beyer*, Jakob Troidl*, Saeed Boorboor,
Markus Hadwiger, Arie Kaufman, and Hanspeter Pfister(*) indicates equal contribution
CONNECTOMICS VIS | STAR | EUROVIS 2022

The Connectome – Discovering the Wiring Diagram of the Brain

CONNECTOMICS VIS | STAR | EUROVIS 2022

Harvard Center for Brain ScienceNeuroscience

Macro

Meso

Nano

Resolution:

Millimeter

Micrometer

Nanometer

Data Size:

1 Byte / 1mm3

1 Gigabyte / 1mm3~2 Petabyte / 1mm3

CONNECTOMICS VIS | STAR | EUROVIS 2022

3Terminology

CONNECTOMICS VIS | STAR | EUROVIS 2022

4

CONNECTOMICS VIS | STAR | EUROVIS 2022

5AxonSynapseDendrite
CONNECTOMICS VIS | STAR | EUROVIS 2022

6

CONNECTOMICS VIS | STAR | EUROVIS 20227

Connectomics Workflow

NeuronalTissue

DataAcquisition

CONNECTOMICS VIS | STAR | EUROVIS 2022

Alignment &Registration

Segmentation& Proofreading

Exploration &Analysis

8

This STAR
Our Focus:•Visualization approaches for every step along
the high-resolution connectomics pipeline•We discuss:•> 200 papers•Open source tools•Public datasets

connectomics-vis-survey.github.io
Website template: @Sarah Schoettler

CONNECTOMICS VIS | STAR | EUROVIS 20229

Connectomics Data••••

Large image stacks
Segmentation image stacks
Connectivity data
Meta data

CONNECTOMICS VIS | STAR | EUROVIS 202210

Connectomics Workflow

Neuronal
Tissue

DataAcquisition

CONNECTOMICS VIS | STAR | EUROVIS 2022

Alignment &Registration

Segmentation& Proofreading

Exploration &Analysis

11Micro- and Nanoscale Brain Imaging•

Light Microscopy• Resolves entire neurons• Resolution: x/y: 200 nm, z: 600 nm•Electron Microscopy• Resolves cell organelles and synapses• Resolution: x/y 3-5 nm, z: 30-50 nm

CONNECTOMICS VIS | STAR | EUROVIS 2022

12

Multi-Beam Acquisition•

Takes multiple images in parallel

•

Here:

•

61 images•

Tissue size: 108 x 108 µm
Data size: 495 MB•

Eberle et al.,J Microscopy 2015

CONNECTOMICS VIS | STAR | EUROVIS 2022

Automated Acquisition of 1mm2 Section•

1mm2: Previously 5 hours, now
less than 5 minutes

Richard Schalek
CONNECTOMICS VIS | STAR | EUROVIS 2022

Visualization for Image Acquisition•Goal:• Quick sanity check of data
• Detecting acquisition errors• Unfocused regions• Dirt• Non-uniform contrast• Beam alignment
Mbeam Viewer

CONNECTOMICS VIS | STAR | EUROVIS 2022

Daniel Haehn
15

CONNECTOMICS VIS | STAR | EUROVIS 2022

Mbeam Viewer
Daniel Haehn et al.
16Connectomics Workflow

NeuronalTissue

DataAcquisition

CONNECTOMICS VIS | STAR | EUROVIS 2022

Alignment &Registration

Segmentation& Proofreading

Exploration &Analysis

17

Alignment & Registration
2D Alignment• Align and stitch individual image tiles in xy plane• 3D Registration• Register slices along z direction•CONNECTOMICS VIS | STAR | EUROVIS 2022

18

Visualization for Alignment & Registration•Goal:• Show alignment/registration
progress to identify errors• Highlight identified image
features (allow modification)• Show result

Mbeam Viewer
CONNECTOMICS VIS | STAR | EUROVIS 2022

Daniel Haehn
19

Registration & Alignment

RHAligner
Haehn et al.

CONNECTOMICS VIS | STAR | EUROVIS 2022

20

Registration & Alignment

RHAligner
Haehn et al.

CONNECTOMICS VIS | STAR | EUROVIS 2022

21

Connectomics Workflow

Neuronal
Tissue

DataAcquisition

CONNECTOMICS VIS | STAR | EUROVIS 2022

Alignment &Registration

Segmentation& Proofreading

Exploration &Analysis

22Segmentation•Labeling of individual structures in neuronal tissue
Synapse annotation
Connectivity extraction•Automatic, semi-automatic, and manual approaches••

CONNECTOMICS VIS | STAR | EUROVIS 2022

23

Visualization for Segmentation•Goal:• Verify and correct segmentation results• Manage large segmentation project•

Approaches:
1. Visualize segmentation result or ongoing segmentation
2. Visual proofreading
3. Visualize segmentation provenance or progress

CONNECTOMICS VIS | STAR | EUROVIS 2022

24

Neuron Tracing in VR[Usher et al., IEEE Vis 2017]

CONNECTOMICS VIS | STAR | EUROVIS 2022

[McDonald et al., IEEE Vis 2020]

25Visualize Segmentation Result

rhoana.org/dojo/CONNECTOMICS VIS | STAR | EUROVIS 2022

26

Neuroglancer (Google)
CONNECTOMICS VIS | STAR | EUROVIS 2022

27

Visual Proofreading

CONNECTOMICS VIS | STAR | EUROVIS 2022

28

Manual Proofreading
Users can manually change
incorrect segmentations• Example tools:• Catmaid• Viking Viewer• WebKnossos• VAST•[Sallfeld et al., Bioinformatics 2009 - Catmaid]
CONNECTOMICS VIS | STAR | EUROVIS 2022

29

(Semi-)Automatic Proofreading
Minimizes required user input• Speeds up and simplifies
proofreading• Example tools:• NeuTu• Dojo• WebKnossos•[Zhao et al., Front. in Neural Circuits 2018 - NeuTu]
CONNECTOMICS VIS | STAR | EUROVIS 2022

30

Example: Dojo – Semi-Automatic Proofreading
Web-based, collaborative proofreading of EM annotations[Haehn et al., IEEE Vis 2014 - Dojo]
CONNECTOMICS VIS | STAR | EUROVIS 2022

31

CONNECTOMICS VIS | STAR | EUROVIS 2022

32

Crowdsourced Proofreading••

Mobilize gamers and citizen scientists to help in proofreading
Users improve existing (automatic) segmentation[Dorkenwald et al., Nature Methods 2022]
CONNECTOMICS VIS | STAR | EUROVIS 2022

33

CONNECTOMICS VIS | STAR | EUROVIS 2022

Eyewire.org
34

Connectomics Workflow

NeuronalTissue

DataAcquisition

CONNECTOMICS VIS | STAR | EUROVIS 2022

Alignment &Registration

Segmentation& Proofreading

Exploration &Analysis

35

Visualization for Exploration & Analysis•Goal:• Visual exploration of connectomics data
• Quantitative analysis• Connectivity analysis• Hypotheses Testing•

Approaches:
1.
Spatial Exploration
2.
Connectivity Exploration
3.
Visualization-focused Analysis

CONNECTOMICS VIS | STAR | EUROVIS 2022

36

Spatial Exploration•

Unsegmented/Segmented Data•

Approaches:• Slice Views• Volume Rendering• Surface Rendering[Google Inc. - Neuroglancer]

CONNECTOMICS VIS | STAR | EUROVIS 2022

37

Scalable EM Volume Visualization[Hadwiger et al., IEEE Vis 2012]

CONNECTOMICS VIS | STAR | EUROVIS 2022

[Google Inc. - Neuroglancer]

38

Confocal Microscopy Visualization[Wan et al., Pacific Vis 2012 - FluoRender]

CONNECTOMICS VIS | STAR | EUROVIS 2022

39

Connectivity Exploration•••

Study connectivity in data (at multiple levels)
Abstract complex 3D morphology
Approaches:• Node-Link views• Matrix views• Abstract connectivity views[Agus et al., Eurovis 2019]

CONNECTOMICS VIS | STAR | EUROVIS 202240

Connectivity Exploration[Clements et al., bioRxiv 2020 - neuPrint]
CONNECTOMICS VIS | STAR | EUROVIS 2022

41

Connectivity Analysis

CONNECTOMICS VIS | STAR | EUROVIS 2022

[Clements et al., bioRxiv 2020 - neuPrint]

42

Connectivity Analysis

CONNECTOMICS VIS | STAR | EUROVIS 2022

[Clements et al., bioRxiv 2020 - neuPrint]

43

Connectivity Analysis

CONNECTOMICS VIS | STAR | EUROVIS 2022

[Clements et al., bioRxiv 2020 - neuPrint]

44

Connectivity Analysis

CONNECTOMICS VIS | STAR | EUROVIS 2022

[Clements et al., bioRxiv 2020 - neuPrint]

45

Connectivity Exploration[Sorger et al., BioVis 2013 - NeuroMAP]
CONNECTOMICS VIS | STAR | EUROVIS 202246

Connectivity Exploration••

Scalable subway map inspired 2D visualization
Linked with original 3D Data

[Al-Awami et al., IEEE Vis 2014 - Neurolines]
CONNECTOMICS VIS | STAR | EUROVIS 2022

47

Visual Encoding: Synapse Links

CONNECTOMICS VIS | STAR | EUROVIS 2022

51

Visualization-Focused Analysis•Approaches:• Neuron Analysis• Visual Query Systems• Neighborhood Analysis

CONNECTOMICS VIS | STAR | EUROVIS 202253

Neuron Analysis••

Visualize & quantify neuron shapes
Compare neurons

CONNECTOMICS VIS | STAR | EUROVIS 2022

54

Neuron Analysis•[Cuntz et al., PLOS Computational Biology 2010]
CONNECTOMICS VIS | STAR | EUROVIS 2022

55

TO

Neuron Analysis[Pastor et al., Applied Sciences 2021]
CONNECTOMICS VIS | STAR | EUROVIS 2022

56

Neuron Analysis
NBLAST: pairwise similarity computation for (fly) neurons. Organizing & searching large datasets
Identifying the same neuron imaged with different
modalities and across hemispheres

Depends on position and local geometry[Image by P. Schlegel, NAVis documentation]

CONNECTOMICS VIS | STAR | EUROVIS 2022

57

Visual Query Systems•••

Interactive analysis of connectomics data
Easy-to-use for domain scientist
Scalable & flexible[Bruckner et al., IEEE Vis 2009]
CONNECTOMICS VIS | STAR | EUROVIS 2022

58

Visual Query Systems

intersectconnected

set

operator

predicate

CONNECTOMICS VIS | STAR | EUROVIS 2022

axons

ROI

dendrite 1[Beyer et al., IEEE Vis 2013]all59

Neighborhood Analysis•••Beyond pure connectivity
Analyze cell organelles andlocal spatial neighborhoods
Examples:• Astrocytes• Glycogen• Mitochondria

CONNECTOMICS VIS | STAR | EUROVIS 2022

Dendrite
Mitochondrion
Synapses61

Neighborhood Analysis - Astrocytes[Mohammed et al., IEEE Vis 2017]
CONNECTOMICS VIS | STAR | EUROVIS 2022

62

astrocyte
dendriteaxon

3D Abstraction of Neurites

ASTROCYTES

abstract

concrete

NEURITESCONNECTOMICS VIS | STAR | EUROVIS 2022

abstract67

Neighborhood Analysis••

Our current EuroVis paper “Barrio: Customizable Spatial
Neighborhood Analysis and Comparison for Nanoscale BrainStructures”
TODO: add figure

CONNECTOMICS VIS | STAR | EUROVIS 2022

69

Datasets & Open-Source Tools••

Commonly used datasets are listed in paper
Links and resources of ready to use open-source tools

H01 dataset
dotMotif

FlyEM Hemibrain dataset… and many many more!!!

CONNECTOMICS VIS | STAR | EUROVIS 2022

70

Future Research: Multimodal Data••••

Optical imaging is limited inresolution
EM can only image dead tissue
How to visualize living tissue?
How can we combine modalities
to get the best of both worlds?
[Velicky et al., bioRxiv 2022]

CONNECTOMICS VIS | STAR | EUROVIS 2022

72

Future Research: Large Connectivity Graph Exploration
Large and dense brain
networks are available now
How to query and (visually)
analyze such networks

CONNECTOMICS VIS | STAR | EUROVIS 2022

NetworkMotif73

Future Research: Scalability
Future (whole brain) connectomics datasets
will be an order of magnitude larger thancurrent data sets
How can we process & visually analyze suchdatasets efficiently?

Image by Sven Dorkenwald

CONNECTOMICS VIS | STAR | EUROVIS 2022

74

Take This With You•

Visual Computing needed in all steps of connectomics pipeline•Challenges:• Scalability• Computational• Visual Complexity• Accessibility

CONNECTOMICS VIS | STAR | EUROVIS 2022

75

Thank You!

connectomics-vis-survey.github.io
CONNECTOMICS VIS | STAR | EUROVIS 2022

76