Open Connectome Project & NeuroData: Enabling Data-Driven Neuroscience at Scale Contact: jovo@jhu.edu References [1]: Burns et al, SSDBM, 2013 [2]: Burns et al, Neuron, 2014 [3]: Gray Roncal et al., Frontiers in Neuroinformatics, 2015 OCP is graciously supported by the following awards: NIBIB 1RO1EB016411-01 (CRCNS), DARPA N66001-14-1-4028 (GRAPHS), NSF ACI-1261715, NSF OCI-1040114, NIDA 1R01DA036400-01, JHU Applied Physics Laboratory IRAD, and JHU Whiting School of Engineering, Dean's Award, Kavli Neuroscience Discovery Institute.. Acknowledgements How do I: - Look at some data? - Show some data to my labmates/colleages? - Download some data to play with on my computer? - Share my data? - Manually annotate some data? - Train my machine vision algorithm on some data? - Run my machine vision algorithm at scale? - Assess the performance of my classifiers? - Infer neuroscientific meaning? Background Challenges Resolution http://neurodata.io - Modalities that now obtain terascale images: - Electron Microscopy - Array Tomography - CLARITY - Calcium Imaging - Expansion Microscopy - X-Ray Microscopy - Multimodal MRI - Loading the entire dataset into memory using MATLAB/R/Python is not possible. - Many neuroscientists want access to the most expensive and difficult datasets to obtain. - In genetics and physics, there exist reference datasets and Web-services that the entire community uses essentially daily. - In neuroscience, we did not yet have reference datasets, or Web-services to enable access and analysis by anybody of interest. Action Action - We built open source Web-services including a neurodata datastore and analytics stack. - We have Ingested >100 terabytes of data across modalities and scales. - We answered scientific questions requiring data-intensive computing, including counting & discovering spatial distributions of various organelles. Challenge - The data are too large to visualize, analyze or often even store on local hardware. - Data management & analytics both need petascale capabilities. - Raw & derived data need to be accessible in ways convenient for analysis. - Different modalities have different needs. Background - Neuroscientists can now routinely collect multi-terabyte neuroimage datasets. - The traditional pattern of storing and analyzing data locally fails for these data. - We need a new computing paradigm that supports analysis and sharing of petascale neuroimages across modalities and scales. Resolution - Many exciting datasets publicly available at http://neurodata.io - Anybody can visualize, download, upload, or analyze neuroimages of any kind/size. - Neurodata analysis at any scale is now trivially reproducible and extensible. - A data standard for massive neuroscience image stacks exists and is extensible. Come to our Booth #2139 Workflow Visualize Images Spatial DB Color Corrected Segments / Skeletons Visualize Overlay Aligned Images NeuroData Analysis Reproduce / Extend Geometric Alignment Machine Vision Ingest Chromatic Alignment Compute Statistics ocpviz ocpviz Jupyter Anno DB Visualize Overlay RAMONify ocpviz Nonlinear Register ocpviz Global Coordinate Legend Local Datastore NeuroDataCloud Cluster/AMI Local Workstation Images Metadata Local Workstation NeuroDataAnalysis Cluster/AMI NeuroDataVision NeuroData Notebooks FlashR FlashGraph Petascale Datastore NeuroDataBase Ingest Cutout OCPviz Image Database Annotation Database Tile Cache Metadata Machine Annotation Geometric Alignment Manual Annotation Chromatic Alignment LONI i2g/m2g CAJAL/ndio RESTful Services RESTful Services CAJAL/ndio Ingested Datasets Computational Ecosystem Example Use Case - If you have big data problems, we might be able to help. - All our software is open source and documented at http://docs.neurodata.io. - We have released an Amazon Marketplace Instance (AMI) to enable running our NeuroData analysis stack without managing a cluster. - All analyzed are performed either in MATLAB or in iPython/Jupyter notebooks. - We are al