Package: igneous-pipeline
Igneous# A few examples. You can also script Igneous. Read on!$igneousimagexfergs://other-lab/datafile://./my-data--queue./xfer-queue--shape2048,2048,64
$igneousimagedownsamplefile://./my-data--mip0--queue./ds-queue
$igneousexecute-x./ds-queue# -x exit when finished$igneousmeshforges3://my-data/seg--mip2--queuesqs://mesh-queue
$igneous--parallel4executesqs://mesh-queue
$igneousskeletonforges3://my-data/seg--mip2--queuesqs://mesh-queue
$igneousskeletonmerges3://my-data/seg--queuesqs://mesh-queue
$igneousexecutesqs://mesh-queue

$igneous--helpIgneous is aTaskQueueand CloudVolume based pipeline for producing and managing visualizable NeuroglancerPrecomputedvolumes. It usesCloudVolumefor accessing data on AWS S3, Google Storage, or the local filesystem. It can operate in the cloud using anSQStask queuing system or run locally on a single machine or cluster (using a file based SQS emulation).Igneous is useful for downsampling, transferring, deleting, meshing, and skeletonizing large images. There are a few more esoteric functions too. You can watch avideo tutorialhere.Originally by Nacho and Will.Pre-Built Docker ContainerYou can use this container for scaling big jobs horizontally or to experiment with Igneous within the container.https://hub.docker.com/r/seunglab/igneous/InstallationYou'll need Python 3, pip, (possibly) a C++ compiler (e.g. g++ or clang), and virtualenv. It's tested under Ubuntu 16.04 and Mac OS Monterey.pipinstalligneous-pipelineManual InstallationSometimes it's useful to tweak tasks for special circumstances, and so you'll want to use a developer installation.gitclonegit@github.com:seung-lab/igneous.gitcdigneous
virtualenvvenvsourcevenv/bin/activate
pipinstallnumpy
pipinstall-rrequirements.txt
pythonsetup.pydevelopIgneous is intended as a self-contained pipeline system and not as a library. Such uses are possible, but not supported. If specific functionality is needed, please open an issue and we can break that out into a library as has been done with several algorithms such astinybrain,zmesh, andkimimaro.Sample Local UseBelow we show three ways to use Igneous on a local workstation or cluster. As an example, we generate meshes for an already-existing Precomputed segmentation volume.In Memory Queue (Simple Execution)This procedure is good for running small jobs as it is very simple, allows you to make use of parallelization, but on the downside it is brittle. If a job fails, you may have to restart the entire task set.fromtaskqueueimportLocalTaskQueueimportigneous.task_creationastc# Mesh on 8 cores, use True to use all corescloudpath='gs://bucket/dataset/labels'tq=LocalTaskQueue(parallel=8)tasks=tc.create_meshing_tasks(cloudpath,mip=3,shape=(256,256,256))tq.insert(tasks)tq.execute()tasks=tc.create_mesh_manifest_tasks(cloudpath)tq.insert(tasks)tq.execute()print("Done!")Filesystem Queue (Producer-Consumer)This procedure is more robust as tasks can be restarted if they fail. The queue is written to the filesystem and as such can be used by any processor that can read and write files to the selected directory. Thus, there is the potential for local cluster processing. Conceptually, a single producer script populates a filesystem queue ("FileQueue") and then typically one worker per a core consumes each task. The FileQueue allows for leasing a task for a set amount of time. If the task is not completed, it recycles into the available task pool. The order with which tasks are consumed is not guaranteed, but is approximately FIFO (a random task is selected from the next 100 to avoid conflicts) if all goes well.This mode is very new, so please report any issues. You can read about the queue designhere. In particular, we expect you may see problems with NFS or other filesystems that have problems with networked file locking. However, purely local use should generally be issue free. You can read more tips on using FileQueuehere. You can remove a FileQueue by deleting its containing directory.Producer ScriptfromtaskqueueimportTaskQueueimportigneous.task_creationastc# Mesh on 8 cores, use True to use all corescloudpath='gs://bucket/dataset/labels'tq=TaskQueue("fq:///path/to/queue/directory")tasks=tc.create_meshing_tasks(cloudpath,mip=3,shape=(256,256,256))tq.insert(tasks)tq.execute()tasks=tc.create_mesh_manifest_tasks(cloudpath)tq.insert(tasks)tq.execute()print("Tasks created!")Consumer ScriptfromtaskqueueimportTaskQueueimportigneous.tasks# magic import needed to provide task definitionstq=TaskQueue("fq:///path/to/queue/directory")tq.poll(verbose=True,# prints progresslease_seconds=600,# allow exclusive 10 min per tasktally=True# makes tq.completed work, logs 1 byte per completed task)Sample Cloud UseIgneous is intended to be used with Kubernetes (k8s). A pre-built docker container is located on DockerHub asseunglab/igneous. A sampledeployment.yml(used withkubectl create -f deployment.yml) is located in the root of the repository.As Igneous is based onCloudVolume, you'll need to create agoogle-secret.jsonoraws-secret.jsonto access buckets located on these services.You'll need to create an Amazon SQS queue to store the tasks you generate. Google's TaskQueue was previously supported but the API changed. It may be supported in the future.Populating the SQS QueueThere's a bit of an art to achieving high performance on SQS. You can read more about ithere.importsysfromtaskqueueimportTaskQueueimportigneous.task_creationastccloudpath=sys.argv[1]# Get qurl from the SQS queue metadata, visible on the web dashboard when you click on it.tq=TaskQueue("sqs://queue-url")tasks=tc.create_downsampling_tasks(cloudpath,mip=0,fill_missing=True,preserve_chunk_size=True)tq.insert(tasks)print("Done!")Executing Tasks in the CloudThe following instructions are for Google Container Engine, but AWS has similar tools.# Create a Kubernetes cluster# e.g.exportPROJECT_NAME=exampleexportCLUSTER_NAME=exampleexportNUM_NODES=5# arbitrary# Create a Google Container Clustergcloudcontainer--project$PROJECT_NAMEclusterscreate$CLUSTER_NAME--zone"us-east1-b"--machine-type"n1-standard-16"--image-type"GCI"--disk-size"50"--scopes"https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.full_control","https://www.googleapis.com/auth/taskqueue","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append"--num-nodes$NUM_NODES--network"default"--enable-cloud-logging--no-enable-cloud-monitoring# Bind the kubectl command to this clustergcloudconfigsetcontainer/cluster$CLUSTER_NAME# Give the cluster permission to read and write to your bucket# You only need to include services you'll actually use.kubectlcreatesecretgenericsecrets\--from-file=$HOME/.cloudvolume/secrets/google-secret.json\--from-file=$HOME/.cloudvolume/secrets/aws-secret.json\--from-file=$HOME/.cloudvolume/secrets/boss-secret.json# Create a Kubernetes deploymentkubectlcreate-fdeployment.yml# edit deployment.yml in root of repo# Resizing the clustergcloudcontainerclustersresize$CLUSTER_NAME--num-nodes=20# arbitrary node countkubectlscaledeploymentigneous--replicas=320# 16 * nodes b/c n1-standard-16 has 16 cores# Spinning down# Important: This will leave the kubernetes master running which you# will be charged for. You can also fully delete the cluster.gcloudcontainerclustersresize$CLUSTER_NAME--num-nodes=0kubectldeletedeploymentigneousCommand Line Interface (CLI)Igneous also comes with a command line interface for performing some routine tasks. We currently supportdownsample,xfer,mesh,skeleton, andexecuteand plan to add more Igneous functions as well. Checkigneous --helpto see the current menu of functions and their options.The principle of the CLI is specify a source layer, a destination layer (if applicable), and aTaskQueue(e.g.sqs://orfq://). First, populate the queue with the correct task type and then execute against it.The CLI is intended to handle typical tasks that aren't too complex. If your task gets weird, it's time to try scripting!igneousimagedownsamplegs://my-lab/data--mip0--queue./my-queue
igneousexecute./my-queue

igneous--helpFor those that have been using Igneous a long time,igneous executecan replacepython igneous/task_execution.py.CapabilitiesYou can find the following tasks inigneous/tasks/tasks.pyand can use them via editing or importing functions fromigneous/task_creation.py.CapabilityTasksDescriptionDownsamplingDownsampleTaskGenerate image hierarchies.MeshingMeshTask, MeshManifestTaskCreate object meshes viewable in Neuroglancer.SkeletonizeSkeletonTask, SkeletonMergeTaskCreate Neuroglancer viewable skeletons using a modified TEASAR algorithm.TransferTransferTaskCopy data, supports rechunking and coordinate translation.DeletionDeleteTaskDelete a data layer.Contrast NormalizationLuminanceLevelsTask, ContrastNormalizationTaskSpread out slice histograms to fill value range.Connected ComponentsCCLFacesTask, CCLEquivalancesTask, more...Compute the 6-way CCL of the whole segmentation.QuantizationQuantizeTaskRescale values into 8-bit to make them easier to visualize.RemappingWatershedRemapTaskRemap segmentations to create agglomerated labels.Eyewire Consensus ImportHyperSquareConsensusTaskMap Eyewire consensus into Neuroglancer.HyperSquare IngestHyperSquareTask(deprecated) Convert Eyewire's HyperSquare format into Precomputed.HyperSquareConsensusHyperSquareConsensusTaskApply Eyewire consensus to a watershed version in Precomputed.Downsampling (DownsampleTask)Requires compiled tinybrain library.For any but the very smallest volumes, it's desirable to create smaller summary images of what may be multi-gigabyte
2D slices. The purpose of these summary images is make it easier to visualize the dataset or to work with lower
resolution data in the context of a data processing (e.g. ETL) pipeline.Image (uint8, microscopy) datasets are typically downsampled in an recursive hierarchy using 2x2x1 average pooling. Segmentation (uint8-uint64, labels) datasets (i.e. human ground truth or machine labels) are downsampled using 2x2x1 mode pooling in a recursive hierarchy using theCOUNTLESS algorithm. This means that mip 1 segmentation labels are exact mode computations, but subsequent ones may not be. Under this scheme, the space taken by downsamples will be at most 33% of the highest resolution image's storage.Whether image or segmentation type downsampling will be used is determined from the neuroglancer info file's "type" attribute.CLI DownsampleHere we show an example where we insert the tasks to downsample 4 mip levels using 2x2x1 pooling into a queue and process it. Then we insert the tasks to downsample from mip 4 up to mip 7 using 2x2x2 downsamples that ignores background to avoid ghosted images when multiple Z are combined.PATH=gs://mydataset/layerQUEUE=fq://./my-queue# could also be sqs://igneousimagedownsample$PATH--mip0--num-mips4--queue$QUEUE# downsample 2x2x1igneousexecute$QUEUE# process the queueigneousimagedownsample$PATH--mip4--num-mips3--volumetric--sparse--queue$QUEUE# superdownsample w/ 2x2x2 sparseigneousimagedownsample$PATH--mip0--queue$QUEUE--sharded# sharded downsampleigneousimagedownsample$PATH--queue$QUEUE--zrange0,1# downsample only a single sliceigneousexecute$QUEUE# process the queueScripting Downsampletasks=create_downsampling_tasks(layer_path,# e.g. 'gs://bucket/dataset/layer'mip=0,# Start downsampling from this mip level (writes to next level up)fill_missing=False,# Ignore missing chunks and fill them with blackaxis='z',num_mips=5,# number of downsamples to produce. Downloaded shape is chunk_size * 2^num_mipchunk_size=None,# manually set chunk size of next scales, overrides preserve_chunk_sizepreserve_chunk_size=True,# use existing chunk size, don't halve to get more downsamplessparse=False,# for sparse segmentation, allow inflation of pixels against backgroundbounds=None,# mip 0 bounding box to downsampleencoding=None# e.g. 'raw', 'compressed_segmentation', etcdelete_black_uploads=False,# issue a delete instead of uploading files containing all backgroundbackground_color=0,# Designates the background colorcompress='gzip',# None, 'gzip', and 'br' (brotli) are optionsfactor=(2,2,1),# common options are (2,2,1) and (2,2,2))# for sharded downsample (only 1 mip at a time)tasks=create_image_shard_downsample_tasks(cloudpath,mip=0,fill_missing=False,sparse=False,chunk_size=None,encoding=None,memory_target=MEMORY_TARGET,agglomerate=False,timestamp=None,factor=(2,2,1))VariableDescriptionlayer_pathLocation of data layer. e.g. 'gs://bucket/dataset/layer'. c.f. CloudVolumemipInteger. Which level of the resolution heirarchy to start downsampling from. 0 is highest res. Higher is lower res. -1 means use lowest res.fill_missingIf a file chunk is missing, fill it with zeros instead of throwing an error.chunk_sizeForce this chunk_size in the underlying representation of the downsamples. Conflicts withpreserve_chunk_sizepreserve_chunk_size(True) Use the chunk size of this mip level for higher downsamples. (False) Use a fixed block size and generate downsamples with decreasing chunk size. Conflicts withchunk_size.sparseOnly has an effect on segmentation type images. False: The dataset contains large continuous labeled areas (most connectomics datasets). Uses theCOUNTLESS 2Dalgorithm. True: The dataset contains sparse labels that are disconnected. Use theStippled COUNTLESS 2Dalgorithm.boundsOnly downsample this region. If using a restricted bounding box, make sure it's chunk aligned at all downsampled mip levels.encodingForce 'raw' or 'compressed_segmentation' for segmentation volumes.delete_black_uploadsIssue a delete instead of uploading files containing all background.background_colorDesignates the background color. Only affectsdelete_black_uploads, notfill_missing.compressWhat compression algorithm to use: None, 'gzip', 'br' (brotli)Data Transfer / Rechunking (TransferTask)A common task is to take a dataset that was set up as single slices (X by Y by 1) chunks. This is often appropriate
for image alignment or other single section based processing tasks. However, this is not optimal for Neuroglancer
visualization or for achieving the highest performance over TCP networking (e.g. withCloudVolume). Therefore, it can make sense to rechunk the dataset to create deeper and overall larger chunks (e.g. 64x64x64, 128x128x32, 128x128x64). In some cases, it can also be desirable to translate the coordinate system of a data layer.TheTransferTaskwill automatically run the first few levels of downsampling as well, making it easier to
visualize progress and reducing the amount of work a subsequentDownsampleTaskwill need to do.Another use case is to transfer a neuroglancer dataset from one cloud bucket to another, but often the cloud
provider's transfer service will suffice, even across providers.CLI TransferHere's an example where we transfer from a source to destination dataset. There are many options available, seeigneous xfer --help.igneousimagexfer$SRC$DEST--queue$QUEUEigneousimagexfer$SRC$DEST--queue$QUEUE--sharded
igneous-p4execute$QUEUEWe have developed some calculation aids to help you pick the right shape for the transfer task.igneousdesignds-shapegs://bucket/dataset--shape1024,1024,64--factor2,2,1
>>>715.8MB

igneousdesignds-memorygs://bucket/dataset3.5e9--verbose
>>>DataWidth:8>>>Factor:(2,2,1)>>>ChunkSize:512,512,16>>>MemoryLimit:3.5GB
>>>-----
>>>OptimizedShape:4096,4096,16
>>>Downsamples:3>>>MemoryUsed*:2.9GB
>>>>>>*memoryusedisforretainingtheimageandalldownsamples.
>>>Additionalcostsmaybeincurredfromprocessing.Scripting Transfertasks=create_transfer_tasks(src_layer_path,dest_layer_path,chunk_size=None,shape=None,fill_missing=False,translate=None,bounds=None,mip=0,preserve_chunk_size=True,encoding=None,skip_downsamples=False,delete_black_uploads=False,background_color=0,agglomerate=False,timestamp=None,compress='gzip',factor=None,sparse=False,dest_voxel_offset=None,memory_target=3.5e9,max_mips=5)# To create a sharded volume from a non-sharded volumetasks=create_image_shard_transfer_tasks(src_layer_path,dst_layer_path,mip=0,chunk_size=None,encoding=None,bounds=None,fill_missing=False,translate=(0,0,0),dest_voxel_offset=None,agglomerate=False,timestamp=None,memory_target=3.5e9,)Most of the options here are the same as for downsample. The major exceptions areshapeandskip_downsamples.shapedesignates the size of a single transfer task and must be chunk aligned. The number of downsamples that will be generated can be computed as log2(shape/chunk_size).skip_downsampleswill prevent downsamples from being generated.Due to the memory constraints, sharded tasks do not automatically generate downsamples.Deletion (DeleteTask)If you want to parallelize deletion of an image layer in a bucket beyond using e.g.gsutil -m rm, you can
horizontally scale out deleting using these tasks. Note that the tasks assume that the information to be deleted
is chunk aligned and named appropriately.CLIigneousimagerm$LAYER--queue$QUEUEigneousexecute$QUEUEScriptingtasks=create_deletion_tasks(layer_path,# data layer to targetmip=0,# Which layer to start deleting fromnum_mips=5,# How many mip levels above to delete (limited by shape)shape=None,# (optional) size of an individual deletion task (must be chunk aligned)bounds=None# Delete only part of a dataset by specifying a cloudvolume.Bbox)Meshing (MeshTask & MeshManifestTask)Requires compiled zmesh library.Meshing is a two stage process. First, the dataset is divided up into a regular grid of tasks that will be meshed independently of
each other using theMeshTask. The resulting mesh fragments are uploaded to the destination layer's meshing directory
(named something likemesh_mip_3_err_40).There are two ways do conduct meshing. The standard "unsharded" way can generate a lot of mesh fragment files. It scales to about 100M labels before it starts incurring unreasonable costs on cloud systems. To handle larger volumes, there is the somwhat more difficult to use sharded meshing process that condenses the number of files by orders of magnitude.Multi-Resolution meshes are supported. Specify the desired number of levels of detail and up to that number will be generated per a mesh (if the mesh is large enough to need it based on the chunk size). Levels of detail are generated by simplifying the base mesh withpyfqmrwith successively more aggressive parameters.Unsharded MeshingWithout additional processing, Neuroglancer has no way of
knowing the names of these chunks (which will be named something like$SEGID:0:$BOUNDING_BOXe.g.1052:0:0-512_0-512_0-512).
The$BOUNDING_BOXpart of the name is arbitrary and is the convention used by igneous because it is convenient for debugging.The manually actuated second stage runs theMeshManifestTaskwhich generates files named$SEGID:0which contains a short JSON snippet like{ "fragments": [ "1052:0:0-512_0-512_0-512" ] }. This file tells Neuroglancer and CloudVolume which mesh files to download when accessing a given segment ID.If multiple levels of detail are specified, the mesh files will be organized differently as they will be using the newer container format.Sharded MeshingSharded Meshes are not only condensed, but also draco encoded with an integer position attribute. The files must be initially meshed and then a set of meshes gathered into the memory of a single machine which can then synthesize the shard file. This requires more time and memory to generate than unsharded meshes, but simplifies management of the resultant data set by creating far fewer files. The shard files have names likea31.shard. A sharded dataset is indicated by the info file in the mesh directory having{ "@type": "neuroglancer_multilod_draco" }.CLI MeshingThe CLI supports only standard Precomputed. Graphene is not currently supported. There are many more options, check outigneous mesh --help,igneous mesh forge --help, andigneous mesh merge --help.# Standard Unsharded Meshing, set nlod > 0 for additional lower levels of detailigneousmeshforge$PATH--mip2--queue$QUEUEigneousexecute$QUEUEigneousmeshmerge$PATH--magnitude2--queue$QUEUE--nlod0igneousexecute$QUEUE# Sharded Meshing, set nlod > 0 for additional lower levels of detailigneousmeshforge$PATH--mip2--queue$QUEUE--shardedigneousexecute$QUEUEigneousmeshmerge-sharded$PATH--queue$QUEUE--nlod0igneousexecute$QUEUEScripting Meshingtasks=create_meshing_tasks(# First Passlayer_path,# Which data layermip,# Which resolution level to mesh at (we often choose near isotropic resolutions)shape=(448,448,448),# Size of a task to mesh, chunk alignment not neededsimplification=True,# Whether to enable quadratic edge collapse mesh simplificationmax_simplification_error=40,# Maximum physical deviation of mesh vertices during simplificationmesh_dir=None,# Optionally choose a non-default location for saving meshescdn_cache=False,# Disable caching in the cloud so updates aren't painful to viewdust_threshold=None,# Don't bother meshing below this number of voxelsobject_ids=None,# Optionally, only mesh these labels.progress=False,# Display a progress bar (more useful locally than in the cloud)fill_missing=False,# If part of the data is missing, fill with zeros instead of raising an errorencoding='precomputed'# 'precomputed' or 'draco' (don't change this unless you know what you're doing)spatial_index=True,# generate a spatial index for querying meshes by bounding boxsharded=False,# generate intermediate shard fragments for later processing into sharded format)tasks=create_mesh_manifest_tasks(layer_path,magnitude=3)# Second PassThe parameters above are mostly self explainatory, but the magnitude parameter ofcreate_mesh_manifest_tasksis a bit odd. What a MeshManifestTask does is iterate through a proportion of the files defined by a filename prefix.magnitudesplits up the work by
an additional 10^magnitude. A high magnitude (3-5+) is appropriate for horizontal scaling workloads while small magnitudes
(1-2) are more suited for small volumes locally processed since there is overhead introduced by splitting up the work.Of note: Meshing is a memory intensive operation. The underlying zmesh library has an optimization for meshing volumes smaller than 1024 voxels on the X and Y dimensions and 512 in Z which could be helpful to take advantage of. Meshing time scales with the number of labels contained in the volume.Skeletonization (SkeletonTask, SkeletonMergeTask)Igneous provides the engine for performing out-of-core skeletonization of labeled images.
The in-core part of the algorithm is provided by theKimimarolibrary.The strategy is to apply Kimimaro mass skeletonization to 1 voxel overlapping chunks of the segmentation and then fuse them in a second pass. Both sharded and unsharded formats are supported. For very large datasets, note that sharded runs better on a local cluster as it can make use ofmmap.We also support computing the cross sectional area at each vertex, but be aware that this will add significant time to the computation (currently many hours for a densely labeled task). This is practical for sparse labeling though. This should be improved substantially in the future.CLI SkeletonizationThe CLI for skeletonization is similar to meshing. Graphene is not supported. However, both sharded and unsharded formats are.# Unsharded Exampleigneousskeletonforge$PATH--mip2--queue$QUEUE--scale2.5--const10igneousexecute$QUEUEigneousskeletonmerge$PATH--queue$QUEUE--tick-threshold500--max-cable-length10000000igneousexecute$QUEUE# Sharded Exampleigneousskeletonforge$PATH--mip2--queue$QUEUE--scale2.5--const10--sharded
igneousexecute$QUEUEigneousskeletonmerge-sharded$PATH--queue$QUEUE--tick-threshold500--max-cable-length10000000--minishard-bits7--shard-bits3--preshift-bits4igneousexecute$QUEUEScripting Skeletonizationimportigneous.task_creationastc# First Pass: Generate Skeletonstasks=tc.create_skeletonizing_tasks(cloudpath,mip,# Which resolution to skeletionize at (near isotropic is often good)shape=Vec(512,512,512),# size of individual skeletonizing tasks (not necessary to be chunk aligned)sharded=False,# Generate (true) concatenated .frag files (False) single skeleton fragmentsspatial_index=False,# Generate a spatial index so skeletons can be queried by bounding boxinfo=None,# provide a cloudvolume info file if necessary (usually not)fill_missing=False,# Use zeros if part of the image is missing instead of raising an error# see Kimimaro's documentation for the below parametersteasar_params={'scale':10,'const':10},object_ids=None,# Only skeletonize these idsmask_ids=None,# Mask out these idsfix_branching=True,# (True) higher quality branches at speed costfix_borders=True,# (True) Enable easy stitching of 1 voxel overlapping tasksdust_threshold=1000,# Don't skeletonize below this physical distanceprogress=False,# Show a progress barparallel=1,# Number of parallel processes to use (more useful locally)spatial_index=True,# generate a spatial index for querying skeletons by bounding boxsharded=False,# generate intermediate shard fragments for later processing into sharded formatcross_sectional_area=False,# Compute the cross sectional area for each vertex.cross_sectional_area_smoothing_window=5,# Rolling average of vertices.)# Second Pass: Fuse Skeletons (unsharded version)tasks=tc.create_unsharded_skeleton_merge_tasks(layer_path,mip,crop=0,# in voxelsmagnitude=3,# same as mesh manifestsdust_threshold=4000,# in nmtick_threshold=6000,# in nmdelete_fragments=False# Delete scratch files from first stage)# Second Pass: Fuse Skeletons (sharded version)tasks=tc.create_sharded_skeleton_merge_tasks(layer_path,# mip is automatically derived from info filedust_threshold=1000,tick_threshold=3500,shard_index_bytes=2**13,minishard_index_bytes=2**15,minishard_index_encoding='gzip',# or Nonedata_encoding='gzip'# or Nonemax_cable_length=None,spatial_index_db=None)Contrast Normalization (LuminanceLevelsTask & ContrastNormalizationTask)Sometimes a dataset's luminance values cluster into a tight band and make the image unnecessarily bright or dark and above all
low contrast. Sometimes the data may be 16 bit, but the values cluster all at the low end, making it impossible to even see without
using ImageJ / Fiji or another program that supports automatic image normalization. Furthermore, Fiji can only go so far on a
Teravoxel or Petavoxel dataset.The object of these tasks are to first create a representative sample of the luminance levels of a dataset per a Z slice (i.e. a frequency count of gray values). This levels information is then used to perform per Z section contrast normalization. In the future, perhaps we will attempt global normalization. The algorithm currently in use reads the levels files for a given Z slice,
determines how much of the ends of the distribution to lop off, perhaps 1% on each side (you should plot the levels files for your own data as this is configurable, perhaps you might choose 0.5% or 0.25%). The low value is recentered at 0, and the high value is stretched to 255 (in the case of uint8s) or 65,535 (in the case of uint16).CLI Contrast Normalization# first pass: create per z-slice histogramigneousimagecontrasthistogam$PATH--queue$QUEUE--coverage0.01--mip0# second pass: apply histogram equalizationigneousimagecontrastequalize$PATH--queue$QUEUE--clip-fraction0.01--mip0Scripting Contrast Normalization# first pass: create per z-slice histogramtasks=create_luminance_levels_tasks(layer_path,coverage_factor=0.01,shape=None,offset=(0,0,0),mip=0)# second pass: apply histogram equalizationtasks=create_contrast_normalization_tasks(src_path,dest_path,shape=None,mip=0,clip_fraction=0.01,fill_missing=False,translate=(0,0,0))Connected Components Labeling (CCL) (Beta!)Igneous supports whole image connected components labeling of a segmentation. Currently, only 6-connected components are supported. The largest image currently supported would have 2^64 voxels (about 18 exavoxels or 18+ whole mouse brains). You can apply CCL to either a labeled image or to a grayscale image that can be binarized with a threshold.The whole image CCL algorithm requires four steps that must be executed in order. The shape specified and optional binarization and dust thresholdsmustbe the same for all steps nonsensical outputs will result. By default, the values will be consistent. To apply a binarization threshold, you can apply both or one of--threshold-lte(<=) and--threshold-gte(>=). You can also apply a--dustthreshold to remove unwanted objects smaller than this threshold.This capability is very new and may have some quirks, so please report any issues.Scripting CCLimportigneous.task_creationastcimportigneous.tasks.image.ccltasks=tc.create_ccl_face_tasks(# Step 1cloudpath,mip,shape=(512,512,512),# optional, for grayscale imagesthreshold_gte=None,threshold_lte=None,dust_threshold=0,)tasks=tc.create_ccl_equivalence_tasks(# Step 2cloudpath,mip,shape,threshold_gte,threshold_lte,dust_threshold)# Step 3# This is just a function to call, not a job to enqueue.# It concentrates the equivalence data onto a single machine# and uploads the relabelings for each grid point.igneous.tasks.image.ccl.create_relabeling(src,mip,shape)tasks=tc.create_ccl_relabel_tasks(# Step 4src_path,dest_path,mip,shape=(512,512,512),chunk_size=None,encoding=None,threshold_gte=None,threshold_lte=None,dust_threshold=0,)CLI CCLigneousimagecclfacesSRC--mip0--queuequeue
igneousexecute-xqueue
igneousimageccllinksSRC--mip0--queuequeue
igneousexecute-xqueue
igneousimagecclcalc-labelsSRC--mip0# computes relabelingigneousimagecclrelabelSRCDEST--mip0--queuequeue--encodingcompresso
igneousexecute-xqueue
igneousimagecclcleanSRC# removes intermediate filesFor smaller images that could reasonably be processed on a single machine there is a shortcutautothat will also automatically execute.igneous-pPARALLELimagecclautoSRCDEST--shape512,512,512--encodingcompresso--queuequeueComputing Per-Object Voxel CountsThis will create aMapBufferdictionary containing the global number of voxels per a label at the location$CLOUDPATH/$KEY/stats/voxel_counts.mb. You can then use this file to lookup the global voxel count for each label.Scripting Voxel Countsimportigneous.task_creationastccloudpath=...mip=0tasks=tc.create_voxel_counting_tasks(cloudpath,mip=mip)tq=LocalTaskQueue(parallel=1)tq.insert_all(tasks)tc.accumulate_voxel_counts(cloudpath,mip)fromcloudfilesimportCloudFilefrommapbufferimportIntMapcf=CloudFile("/".join(cloudpath,"stats","voxel_counts.im"))# for (slow) remote access w/o having to download the file# only works if file is uncompressed on remoteim=IntMap(cf)# for fast local access, but downloads the whole fileim=IntMap(cf.get())im[label]# fetches voxel count for labelCLI Voxel CountsigneousimagevoxelscountSRC--mip0--queuequeue
igneousexecute-xqueue
igneousimagevoxelssumSRC--mip0# no execution neededReordering Z-SlicesWhen acquiring a new microscopy stack, for a variety of reasons,
such as process interruptions, reimaging, etc, the montaged slices
may not be compact in Z or otherwise out of order. Using a JSON
file, you can specify (sparsely) which movements in Z are required
to put the image stack in order.// mapping.json for the 5 slices from 0 to 4 inclusive{"1":2,"2":3,"3":1}// This will result in the order: [ 0, 3, 1, 2, 4 ]The mapping file will be analyzed to ensure no slices are dropped
before creating the task set.CLI Z-Slice ReorderigneousimagereorderSRCDEST--queuequeue--mip0--mapping-filemapping.jsonTissue ROI DetectionSometimes, especially during the alignment of a new microscopy stack, volume bounds may greatly exceed the tissue containing regions. This results in wasted computation when processing large volumes. As of version 8.22.0, CloudVolume can use a set of pre-computed bounding boxes to avoid issuing network requests to empty regions.Igneous can compute these regions by checking low resolution images of the dataset for the presence of tissue and record those bounding boxes in the highest resolution scale of the info file in the following format where the variable names are integers and the bounds are inclusive."rois": [ [ xstart, ystart, zstart, xend, yend, zend ], ... ]The below function performs all computation in one step and does not require using task queues. It can be memory intensive if the volume is large and not sufficiently downsampled. The lowest resolution downsample available will be used, and possibly downsampled further in memory before being analyzed.tc.compute_rois(cloudpath:str,progress:bool=False,suppress_faint_voxels:int=0,dust_threshold:int=10,max_axial_length:int=512,z_step:Optional[int]=None,)When the function finishes executing, it will print out the number of bounding boxes found. Depending on your data, a reasonable number of bounding boxes are between 1 to 15. Above 25 bounding boxes, CloudVolume may incur more than 1 millisecond of additional processing per a cutout.If you see hundreds of bounding boxes have generated unexpectedly, try examining your image more carefully and consider suppressing faint voxels or tiny connected components.CLI Tissue ROI Detectionigneousimageroi$PATH# process whole dataset in one shot# to generate bboxes in bundles of 100 z-slices. This can be helpful# if the tissue region drifts across the image plane as that could# generate a single bounding box nearly equivalent to the entire volume.igneousimageroi$PATH--z-step100# If your image is huge and deep, for example 10,000 z-slices, then# simply retaining it in memory may be challenging as 512 * 512 * 10,000# is 2.6 GVx and image processing may use several multiples of# the input size. You can ask for additional downsampling by specifying# a smaller side image length.igneousimageroi$PATH--max-axial-len256ConclusionIt's possible something has changed or is not covered in this documentation. Please readigneous/task_creation/andigneous/tasks/for the most current information.Please post an issue or PR if you think something needs to be addressed.Related Projectstinybrain- Downsampling code for images and segmentations.kimimaro- Skeletonization of dense volumetric labels.zmesh- Mesh generation and simplification for dense volumetric labels.CloudVolume- IO for images, meshes, and skeletons.python-task-queue- Parallelized dependency-free cloud task management.DracoPy- Encode/Decode Draco compressed meshes in PythonMapBuffer- Zero decode random access to uint64 keyed byte streams (useful for shard construction)AcknowledgementsSpecial thanks to everyone that has contributed to Igneous! I'll fill this section in more fully later, but in particular recent thanks to Jeremy Maitin-Shepard, David Ackermann, Hythem Sidky, and Sridhar Jagannathan for giving advice and sharing code for producing sharded multi-res meshes. Thanks to Chris Roat for improving the CLI's typing. Thanks to Nico Kemnitz for improving the original mesh task. Of course, thanks to lab alumni Ignacio Tartavull who started the project with me and provided its initial impetus.