Package: chunkflow
ProblemPetabyte scale 3D image processing is slow and computationally demanding;Computation has to be distributed with linear scalability;Local cluster and public cloud computing are not fully used at the same time;Duplicated code across a variety of routine tasks is hard to maintain.FeaturesComposableoperators. The chunk operators could be composed in a command line for flexible usage.Hybrid Cloud Distributedcomputation in both local and cloud computers. The task scheduling frontend and computationally heavy backend are decoupled using AWS Simple Queue Service. The backend could be any computer with an internet connection and cloud authentication. Benefit from the robust design, the cheap unstable instances (preemptable intance in Google Cloud, spot instance in AWS) could be used to reduce cost by about threefold!Petabytescale. We have used chunkflow to output over eighteen-petabyte images and scaled up to 3600 nodes with NVIDIA GPUs across three regions inGoogle Cloud, and chunkflow is still reliable.Operators work with3Dimage volumes.You canplug inyour own code as an operator.Check out theDocumentationforinstallationandusage. Try it out by following thetutorial.Image Segmentation ExamplePerform Convolutional net inference to segment 3D image volume with one single command!#!/bin/bashchunkflow\load-tif--file-namepath/of/image.tif-oimage\inference--convnet-modelpath/of/model.py--convnet-weight-pathpath/of/weight.pt\--input-patch-size20256256--output-patch-overlap46464--num-output-channels3\-fpytorch--batch-size12--mask-output-chunk-iimage-oaffs\plugin-fagglomerate--threshold0.7--aff-threshold-low0.001--aff-threshold-high0.9999-iaffs-oseg\neuroglancer-iimage,affs,seg-p33333-v3066you can see your 3D image and segmentation directly inNeuroglancer!Composable OperatorsAfter installation, You can simply typechunkflowand it will list all the operators with help message. We keep adding new operators and will keep it update here. For the detailed usage, please checkout ourDocumentation.Operator NameFunctionadjust-bboxadjust the corner offset of bounding boxaggregate-skeleton-fragmentsMerge skeleton fragments from chunkschannel-votingVote across channels of semantic mapcleanupremove empty files to clean up storagecloud-watchRealtime speedometer in AWS CloudWatchconnected-componentsThreshold the boundary map to get a segmentationcopy-varCopy a variable to a new namecreate-chunkCreate a fake chunk for easy testcreate-infoCreate info file of Neuroglancer Precomputed volumecrop-marginCrop the margin of a chunkdelete-chunkDelete chunk in task to reduce RAM requirementdelete-task-in-queueDelete the task in AWS SQS queuedownsample-uploadDownsample the chunk hierarchically and upload to volumedownload-meshDownload meshes from Neuroglancer Precomputed volumeevaluate-segmentationCompare segmentation chunksfetch-task-from-fileFetch task from a filefetch-task-from-sqsFetch task from AWS SQS queue one by onegenerate-tasksGenerate tasks one by onegaussian-filter2D Gaussian blurring operated in-placeinferenceConvolutional net inferenceload-synapsesLoad synapses from a filesave-synapsesSave synapses as a HDF5 file.save-pointsSave point cloud as a HDF5 file.log-summarySummary of logsmark-completemark task completion as an empty filemaskBlack out the chunk based on another mask chunkmask-out-objectsMask out selected or small objectsmultiplyMultiply chunks with another chunkmeshBuild 3D meshes from segmentation chunkmesh-manifestCollect mesh fragments for objectneuroglancerVisualize chunks using neuroglancernormalize-contrast-nkemNormalize image contrast using histogramsnormalize-intensityNormalize image intensity to -1:1normalize-section-shangNormalization algorithm created by ShangpluginImport local code as a customized operator.quantizeQuantize the affinity mapload-h5Read HDF5 filesload-npyRead NPY filesload-jsonRead JSON filesload-pngsRead png filesload-precomputedCutout chunk from a local/cloud storage volumeload-tifRead TIFF filesload-nrrdRead NRRD filesload-zarrRead Zarr filesremap-segmentationRenumber a serials of segmentation chunkssetup-envPrepare storage infor files and produce tasksskeletonizeCreate centerlines of objects in a segmentation chunkskip-taskIf a result file already exists, skip this taskskip-all-zeroIf a chunk has all zero, skip this taskskip-noneIf an item in task is None, skip this taskthresholdUse a threshold to segment the probability mapviewAnother chunk viewer in browser using CloudVolumesave-h5Save chunk as HDF5 filesave-pngsSave chunk as a serials of png filessave-precomputedSave chunk to local/cloud storage volumesave-tifSave chunk as TIFF filesave-nrrdSave chunk as NRRD fileAffiliationThis package is developed at Princeton University and Flatiron Institute.ReferenceWe have apaperfor this repo:@article{wu_chunkflow_2021,title={Chunkflow: hybrid cloud processing of large {3D} images by convolutional nets},issn={1548-7105},shorttitle={Chunkflow},url={https://www.nature.com/articles/s41592-021-01088-5},doi={10.1038/s41592-021-01088-5},journal={Nature Methods},author={Wu, Jingpeng and Silversmith, William M. and Lee, Kisuk and Seung, H. Sebastian},year={2021},pages={1--2}}