Package: daisy
Daisy: A Blockwise Task SchedulerBlockwise task scheduler for processing large volumetric dataWhat is Daisy?Daisy is a library framework that facilitates distributed processing of big nD datasets across clusters of computers.
It combines the best of MapReduce/Hadoop (the ability to map a process function across elements) and Luigi (the ability to chain dependent tasks together) together in one lightweight and efficient package with a focus on processing nD datasets.Daisy documentations are athttps://daisy-docs.readthedocs.io/OverviewDeveloped by researchers at HHMI Janelia and Harvard, the intention behind Daisy was to develop a scalable and fast distributed block-wise scheduler for processing very large (TBs to PBs) 3D/4D bio image datasets.
We needed a fast and scalable scheduler but also resilient to failures and recoverable/resumable from hardware errors.
Daisy should also be generalizable enough to support efficient processing of different tasks with different computation and input/output modalities.Daisy is lightweightDaisy uses high performance TCP/IP libraries for communications between the scheduler and workers.It minimizes network overheads by sending only coordinates and status checks. Daisy does not enforce the exact method of data transfers to/between workers so that maximum performance is achieved for different tasks.Daisy's API is easy-to-use and extensibleBuilt on Python, Daisy provides an easy-to-use native interface for Python scripts useful for both simple and complex use cases.Simplistically, Daisy is a framework formapping a function across independent sub-blocks in the dataset.More complex usages include specifying inter-block dependencies, inter-task dependencies, using Daisy's array interface and geometric graph interface.Daisy chains complex pipelines of tasksInspired by powerful workflow management frameworks likeLuigifor automating long running tasks and decreasing overall processing time through task pipelining, Daisy allows user to specify dependency between tasks, allowing for task chaining and running multiple tasks in a pipeline with dynamic concurrent per-block execution.For instance, Daisy can chain amaptask and areducetask to implement amap-reducetask for nD datasets. Of course, any other combinations ofmapandreducetasks are composable.By tracking dependencies at theblocklevel, tasks can be executed concurrently to maximize pipelining parallelism.Daisy is tuned for processing datasets with real-world unitsDaisy has a native inferface to represent of regions in a volume in real world units, easily handling voxel sizes, with convenience functions for region manipulation (intersection, union, growing or shrinking, etc.)Daisy provides an array interface that can wrap any object with a shape attribute and slicing operations (zarr, hdf5, numpy array)Daisy provides a geometric graph interface with support for file system and MongoDB storage (nodes are locations in space, and can be accessed by region)Installationpipinstall-egit+https://github.com/funkelab/daisy#egg=daisyAlternatively, install through PyPI (though outdated at the moment):pipinstalldaisyQuickstartSee the following code in aIPython notebook!Map taskFirst, let's run a simplemaptask with Daisy. Supposed we have an arrayathat we want to compute the square for each element and store inbimportnumpyasnpshape=4096000a=np.arange(shape,dtype=np.int64)b=np.empty_like(a,dtype=np.int64)print(a)# prints [0 1 2 ... 4095997 4095998 4095999]We can use the followingprocess_fn:defprocess_fn():# iterating and squaring each element in a and store to bwithnp.nditer([a,b],op_flags=[['readonly'],['readwrite']])asit:withit:forx,yinit:y[...]=x**2%timeitprocess_fn()# 3.55 s ± 22.7 ms per loopprint(b)# prints [0 1 4 ... 16777191424009 16777199616004 16777207808001]Sinceprocess_fnlinearly processesain a single-thread, it is quite slow.
Let's use Daisy to breakainto blocks and runprocess_fnin parallel.First, we'll wrapain adaisy.Arrayand make abarray based onzarrthat multiple concurrent process can write to. We will also defineblock_shape- the granularity that each worker will be working at.importdaisyimportzarrshape=4096000block_shape=1024*16# input array is wrapped in daisy.Array for easy of Roi indexinga=daisy.Array(np.arange(shape,dtype=np.int64),roi=daisy.Roi((0,),shape),voxel_size=(1,))# to parallelize across processes, we need persistent read/write arrays# we'll use zarr here to do do thatb=zarr.open_array(zarr.TempStore(),'w',(shape,),chunks=(block_shape,),dtype=np.int64)# output array is wrapped in daisy.Array for easy of Roi indexingb=daisy.Array(b,roi=daisy.Roi((0,),shape),voxel_size=(1,))Theprocess_fnis then modified slightly to take in ablockobject and perform read/write using the ROIs given by it.# same process function as previously, but with additional code# to read and write data to persistent arraysdefprocess_fn_daisy(block):a_sub=a[block.read_roi].to_ndarray()b_sub=np.empty_like(a_sub)withnp.nditer([a_sub,b_sub],op_flags=[['readonly'],['readwrite']],)asit:withit:forx,yinit:y[...]=x**2b[block.write_roi]=b_subNext, we definetotal_roibased on total amount of work (shape) andblock_roibased on scheduling block size (block_shape). We then make adaisy.Taskand run it.total_roi=daisy.Roi((0,),shape)# total ROI to map process overblock_roi=daisy.Roi((0,),(block_shape,))# block ROI for parallel processing# creating a Daisy task, note that we do not specify how each# worker should read/write to input/output arraystask=daisy.Task(total_roi=total_roi,read_roi=block_roi,write_roi=block_roi,process_function=process_fn_daisy,num_workers=8,task_id='square',)daisy.run_blockwise([task])'''prints Execution Summary-----------------Task square:num blocks : 250completed ✔: 250 (skipped 0)failed    ✗: 0orphaned  ∅: 0all blocks processed successfully'''# %timeit daisy.run_blockwise([task])  # 1.26 s ± 16.1 ms per loopprint(b.to_ndarray())# prints [0 1 4 ... 16777191424009 16777199616004 16777207808001]See that with just a minor modification, using Daisy to run multiple workers in parallel results in a 2.8176x speedups on a computer with 6 cores. For longer running tasks with larger block sizes (to minimize process spawning/joining overheads) the speedups should approach the # of threads/cores running in parallel more.Reduce taskNow we'll write and run areducetask. This task performs a sum of blocks of shapereduce_shapefromband stores the results toc.importmultiprocessingreduce_shape=shape/16# while using zarr with daisy.Array can be easier to understand and less error prone, it is not a requirement.# Here we make a shared memory array for collecting results from different workersc=multiprocessing.Array('Q',range(int(shape/reduce_shape)))defprocess_fn_sum_reduce(block):b_sub=b[block.write_roi].to_ndarray()s=np.sum(b_sub)# compute c idx based on block offset and shapeidx=(block.write_roi.offset/block.write_roi.shape)[0]c[idx]=stotal_roi=daisy.Roi((0,),shape)# total ROI to map process overblock_roi=daisy.Roi((0,),reduce_shape)# block ROI for parallel processingtask1=daisy.Task(total_roi=total_roi,read_roi=block_roi,write_roi=block_roi,process_function=process_fn_sum_reduce,num_workers=8,task_id='sum_reduce',)daisy.run_blockwise([task1])print(c[:])This concludes our quickstart tutorial. For more examples/tutorials please see theexamples/directory.Citing DaisyTo cite this repository please use the following bibtex entry:@software{daisy2022github,
  author = {Tri Nguyen and Caroline Malin-Mayor and William Patton and Jan Funke},
  title = {Daisy: block-wise task dependencies for luigi.},
  url = {https://github.com/funkelab/daisy},
  version = {1.0},
  year = {2022},
}In the above bibtex entry, the version number is intended to be that fromdaisy/setup.py, and the year corresponds to the project's 1.0 release.