Package: daskqueue
daskqueuedaskqueueis small python library built on top of Dask and Dask Distributed that implements a very lightweight Distributed Task queue.Think of this library as a simpler version of Celery built entirely on Dask. Running Celery on HPC environnment (for instance) is usually very tricky whereas spawning a Dask Cluster is a lot easier to manage, debug and cleanup.MotivationDaskis an amazing library for parallel computing written entirely in Python. It is easy to install and offer both a high level API wrapping common collections (arrays, bags, dataframes) and a low level API for written custom code (Task graph with Delayed and Futures).For all its greatness, Dask implements a central scheduler (basically a simple tornado eventloop) involved in every decision, which can sometimes create a central bottleneck. This is a pretty serious limitation when trying use Dask in high throughput situation. A simple Task Queue is usually the best approach when trying to distribute millions of tasks.Thedaskqueuepython library leveragesDask Actorsto implement distributed queues with a simple load balancerQueuePooland aConsummerclass to consumme message from these queues.We used Actors because:Actors are stateful, they  can hold on to and mutate state. They are allowed to update their state in-place, which is very useful when spawning distributed queues !NO CENTRAL SCHEDULING NEEDED :Operations on actors do not inform the central scheduler, and so do not contribute to the 4000 task/second overhead. They also avoid an extra network hop and so have lower latencies. Actors can communicate between themselves in a P2P manner, which makes it pretty neat when having a huge number of queues and consummers.Note:  Dask provides aQueue implementationbut they are mediated by the central scheduler, and so they are not ideal for sending large amounts of data (everything you send will be routed through a central point) and add additionnal overhead on the scheduler when trying to put millions of tasks.Installdaskqueue requires Python 3.6 or newer.
You can install manually by cloning the repository:$pipinstalldaskqueueUsageThis simple example show how to copy files in parallel using Dask workers and a distributed queue:fromdistributedimportClientfromdaskqueueimportQueuePool,ConsumerPoolfromdaskqueue.utilsimportloggerdefprocess_item():returnsum(i*iforiinrange(10**8))if__name__=="__main__":client=Client(n_workers=5,# task function doesn't release the GILthreads_per_worker=1,direct_to_workers=True,)## Paramsn_queues=1n_consumers=5queue_pool=QueuePool(client,n_queues=n_queues)consumer_pool=ConsumerPool(client,queue_pool,n_consumers=n_consumers)consumer_pool.start()foriinrange(5):queue_pool.submit(process_item)# Wait for all work to be doneconsumer_pool.join()## Get resultsresults=consumer_pool.results()Take a look at theexamples/folder to get some usage.ImplementationYou should think of daskqueue as a very simple distributed version of aiomultiprocessing. We have these basic classes:Queue: The daskqueue library provides two queue types :TransientQueue: The default queue class. The submitted messages are appended to an in memory FIFO queue.DurableQueue: This is a disk backed queue for persisting the messages. The tasks are served in FIFO manner. Durable queues append serialized message to a fixed-sized file calledLogSegment. The durable queues also append queue operations to and anIndexSegment. The index segment is a combination of abitcaskindex for segment offsets and WAL file : it is an append only file where we record message status after each queue operation (ready, delivered, acked and failed) and an offset to the message in on of theLogSegmentsQueuePoolActor: Basic Pool actor, it holds a reference to queues and their sizes. It interfaces with the Client and the Consummers. The QueuePool implements a round robin batch submit.ConsumerBaseClass: Abstract class interfaces implementing all the fetching logic for you worker. The Consumers have astart()method where we run an unfinite fetch loop to pop items from queue assigned by QueuePool. The consumer directly communicate with the Queue, providing highly scalable workflows. The Consummer will then get an item from the queue and scheduleprocess_itemon the dask worker's ThreadPoolExecutor, freeing the worker's eventloop to communicate with the scheduler, fetch tasks asynchronously etc ....Performance and LimitationsBenchmarksThedaskqueuelibrary is very well suited for IO bound jobs: by running multiple consummers and queues, communication asynchronously, we can bypass the dask scheduler limit and process **millions of tasks 🥰 !! **The example copy code above was ran on cluster of 20 consummers and 5 queues. The tasks ran are basic file copy between two location (copying form NFS filer). We copied 200 000 files (~ 1.1To) without ever breaking a sweat !We can clearly see the network saturation:Looking at the scheduler metrics, we can have a mean of 19.3%You can take a look at thebenchmark/directory for various benchmarks ran usingdaskqueuevsdask:We put  1_000_000 tasks using dask cluster (2 nodes- 1 thread per process- 4 queues- 8 consumers)The tasks were chunked using  into 1000 calls of 1000 tasks per batchThe client submits to the QueuePool manager usingThe function is 'empty' : just passes and doesn't use CPU or IOProcessing 1_000_000 empty tasks took 338s = 5min36s 😸!!ThroughputFor durable queues, we can achive the following throughput with 1 consumer and 1 queue (running on the same machine)1 queue | 1 consumer :Mean write ops [1tests] 10629.40 wop/sMean read ops [1tests] 11612.37 rop/s5 queues | 5 consumers, we have a near linear speed up for consumers, reader:Mean write ops [1tests] 14430.91 wop/sMean read ops [1tests] 42791.44 rop/sLimitationsAs for the limitation, given the current implementation, you should be mindfull of the following limitations (this list will be updated regularly):We run the tasks in the workers ThreadPool, we inherit all the limitations that the standard dask.submit method have.Task that require multiprocessing/multithreading within a worker cannot be scheduled at the time. This is also true for dask tasks.The QueuePool implement simple scheduling on put and get. More sophisticated scheduling will be implementing in the future.Features roadmapConsumer should run arbitrary funcs (ala celery)Use Worker's thread pool for long running tasks ( probe finished to get results)Wrap consummers in a Consummers classImplement a Distributed Join to know when to stop clusterImplement aconcurrency_limitas the maximum number of active, concurrent jobs each worker process will pick up from its queue at once.Run tasks on custom Worker's executorsAdd benchmarksTestsImplement durable queues with bitcask indexImplement Ack MechanismReschedule Unacked MessageImplement health check mechanismImplement tasks retriesNotify dask dahboardContributingContributions are what makes the open-source community such an amazing place to learn, inspire, and create.
This project is still very very rough! Any contributions you make will benefit everybody else and are greatly appreciated  😍 😍 😍 !Please try to create bug reports that are:Reproducible. Include steps to reproduce the problem.Specific. Include as much detail as possible: which version, what environment, etc.Unique. Do not duplicate existing opened issues.Scoped to a Single Bug. One bug per issue.ReleasingReleases are published automatically when a tag is pushed to GitHub.gitcheckoutmaster
gitpull# Set next version numberexportRELEASE=x.x.x# Create tagsgitcommit--allow-empty-m"Release$RELEASE"gittag-a$RELEASE-m"Version$RELEASE"# Pushgitpushupstream--tagsLicensedaskqueueis copyrightAmine Dirhoussi, and licensed under
the MIT license.  I am providing code in this repository to you under an open
source license.  This is my personal repository; the license you receive to
my code is from me and not from my employer. See theLICENSEfile for details.