Package: task-queue
python-task-queueA client and system for generating, uploading, leasing, and executing dependency free tasks both locally and in the cloud using AWS SQS or on a single machine or cluster with a common file system using file based queues. Of note, file queue requires no setup or queue service and can be used in a distributed fashion on a network filesystem.The task queue uses JSON as a messaging medium, so be aware that e.g. integer dictionary keys can be turned into strings when bound as a parameter.Installationpipinstallnumpy# make sure you do this first on a seperate linepipinstalltask-queueThe task queue uses your CloudVolume secrets located in$HOME/.cloudvolume/secrets/. When using AWS SQS as your queue backend, you must provide$HOME/.cloudvolume/secrets/aws-secret.json. See theCloudVolumerepo for additional instructions.UsageAs of version 2.7.0, there are two ways to create a queueable task. The new way is simpler and probably preferred.MacOS Only: Note that proxy servers are disabled for parallel operation due to libdispatch being not fork-safe.New School: Queueable FunctionsDesignate a function as queueable using the@queueabledecorator. Currently variable positional arguments (*args) and variable keyword arguments (**kwargs) are not yet supported. If a function is not marked with the decorator, it cannot be executed via the queue.fromtaskqueueimportqueueable@queueabledefprint_task(txt):print(str(txt))You then create queueable instantiations of these functions by using the standard librarypartialfunction to create a concrete binding.fromfunctoolsimportpartialbound_fn=partial(print_task,txt="hello world")Old School: RegisteredTask SubclassesDefine a class that inherits from taskqueue.RegisteredTask and implements theexecutemethod. RegisteredTasks contain logic that will render their attributes into a JSON payload and can be reconstituted into a live class on the other side of a task queue.Tasks can be loaded into queues locally or in the cloud and executed later. Here's an example implementation of a trivialPrintTask. The attributes of your container class should be simple values that can be easily encoded into JSON such as ints, floats, strings, and numpy arrays. Let theexecutemethod download and manipulate heavier data. If you're feeling curious, you can see what JSON a task will turn into by callingtask.payload().fromtaskqueueimportRegisteredTaskclassPrintTask(RegisteredTask):def__init__(self,txt=''):super(PrintTask,self).__init__(txt)# attributes passed to super().__init__ are automatically assigned# use this space to perform additional processing such as:self.txt=str(txt)defexecute(self):ifself.txt:print(str(self)+": "+self.txt)else:print(self)Local UsageFor small jobs, you might want to use one or more processes to execute the tasks.fromfunctoolsimportpartialfromtaskqueueimportLocalTaskQueuefrommylibraryimportPrintTask# mylibrary is wherever you defined PrintTasktq=LocalTaskQueue(parallel=5)# use 5 processestasks=(PrintTask(i)foriinrange(2000))# OLD SCHOOLtasks=(partial(print_task,i)foriinrange(2000))# NEW SCHOOLtq.insert_all(tasks)# performs on-line execution (naming is historical)# alterternative serial modeltq.insert(tasks)tq.execute()# delete taskstq.delete(tasks)tq.purge()# delete all tasksThis will load the queue with 1000 print tasks then execute them across five processes.Cloud and Cluster UsageSet up an SQS queue and acquire an aws-secret.json that is compatible with CloudVolume. Generate the tasks and insert them into the cloud queue.You can alternatively set up a file based queue that has the same time-based leasing property of an SQS queue.IMPORTANT: You must import the tasks that will be executed, otherwise the code to execute them has not been loaded.# import gevent.monkey# gevent.monkey.patch_all(thread=False)fromtaskqueueimportTaskQueuefrommylibraryimportPrintTask# mylibrary is wherever you defined PrintTask# region is SQS specific, green means cooperative threadingtq=TaskQueue('sqs://queue-name',region_name="us-east1-b",green=False)tq=TaskQueue('fq:///path/to/queue/directory/')# file queue ('fq')# insert accepts any iterabletq.insert((PrintTask(i)foriinrange(1000)))# OLD SCHOOLtq.insert((partial(print_task,i)foriinrange(1000)))# NEW SCHOOLtq.enqueued# approximate number of tasks in the queue# FileQueue Only:tq.inserted#  total number of tasks insertedtq.completed# number of tasks completed, requires tally=True with polltq.rezero()# reset statistics like inserted and completedtq.release_all()# set all tasks to availableThis inserts 1000 PrintTask JSON descriptions into your SQS queue.Somewhere else, you'll do the following (probably across multiple workers):fromtaskqueueimportTaskQueueimportMY_MODULE# MY_MODULE contains the definitions of RegisteredTaskstq=TaskQueue('sqs://queue-name')tq.poll(lease_seconds=int(LEASE_SECONDS),verbose=True,# print out a success messagetally=True,# count number of tasks completed (fq only!))Poll will check the queue for a new task periodically. If a task is found, it will execute it immediately, delete the task from the queue, and request another. If no task is found, a random exponential backoff of up to 120sec is built in to prevent workers from attempting to DDOS the queue. If the task fails to complete, the task will eventually recirculate within the queue, ensuring that all tasks will eventually complete provided they are not fundementally flawed in some way.Local Container testingIf there is a AWS compatible queue running on a local cluster, e.g.alpine-sqs, the underlying connection client
needs additional parameters. These can be passed into the TaskQueue constructor.The following code on a worker will work in local and production contexts:queue=os.environ['SQS_QUEUE']# for local, set to "default"region_name=os.environ.get('SQS_REGION_NAME')# set only for prodendpoint_url=os.environ.get('SQS_ENDPOINT_URL')# set only for localtqueue=taskqueue.TaskQueue(f'sqs://{queue}',region_name=region_name,endpoint_url=endpoint_url)Example docker-compose.yml for local testing:version:"3.7"services:worker:image:yourlab/yourworker:v1environment:-SQS_QUEUE=default-SQS_ENDPOINT_URL=http://local_sqs:9324depends_on:-local_sqslocal_sqs:image:roribio16/alpine-sqsExample docker-compose.yml for production:version:"3.7"services:worker:image:yourlab/yourworker:v1environment:-SQS_QUEUE=my-aws-queue-SQS_REGION=us-west-1Notes on File Queue# FileQueue Specific Featurestq.inserted# number of inserted taskstq.completed# number of completed tasks (counts rerun tasks too)tq.rezero()# sets tq.inserted and tq.completed to zero.tq.release_all()# sets all tasks to availableFileQueue (fq://) is designed to simulate the timed task leasing feature from SQS and exploits a common filesystem to avoid requiring an additional queue server. You can read in detail about its designon the wiki.There are a few things FileQueue can do that SQS can't and also some quirks you should be aware of. For one, FileQueue can track the number of task completions (tq.completions,tq.poll(..., tally=True)), but it does so by appending a byte to a file calledcompletionsfor each completion. The size of the file in bytes is the number of completions. This design is an attempt to avoid problems with locking and race conditions. FileQueue also tracks insertions (tq.insertions) in a more typical way in aninsertionsfile. Also unlike SQS, FileQueue allows listing all tasks at once.FileQueue also allows releasing all current tasks from their leases, something impossible in SQS. Sometimes a few tasks will die immediately after leasing, but with a long lease, and you'll figure out how to fix them. Instead of starting over or waiting possibly hours, you can set the queue to be made available again (tq.release_all()).As FileQueue is based on the filesystem, it can be managed somewhat via the command line. To delete a queue, justrm -r $QUEUE_PATH. To reset a counter:rm $QUEUE_PATH/completions(e.g.). If you are brave, you could even use themvcommand to reassign a task's availability.We also discovered that FileQueues are also amenable to fixing problems on the fly. In one case, we generated a set of tasks that took 4.5 hours of computation time and decided to run those tasks on a different cluster. The 500k tasks each contained a path to the old storage cluster. Usingfind,xargs, andsedwe were able to fix them efficiently.BundledptqCLI ToolAs of 2.5.0, we now bundle a command line toolptqto make managing running FileQueues easier.ptqstatusfq://./my-queue# prints vital statisticsptqreleasefq://./my-queue# releases all tasks from their leaseptqrezerofq://./my-queue# resets statistics to zeroptqcpfq://./my-queuesqs://my-cloud-queue# copy a queue (no copies of sqs)ptqmvsqs://my-cloud-queuefq://./my-queue# move a queue (all supported)MotivationDistributed dependency free task execution engines (such asIgneous) often make use of cloud based queues like Amazon Simple Queue Service (SQS). In the connectomics field we process petascale images which requires generating hundreds of thousands or millions of cloud tasks per a run. In one case, we were processing serial blocks of a large image where each block depended on the previous block's completion. Each block's run required the generation and upload of millions of tasks and the use of thousands of workers. The workers would rapidly drain the task queue and it was important to ensure that it could be fed fast enough to prevent starvation of this enormous cluster.There are a few strategies for accomplishing this. One way might be to use a fully featured DAG supporting engine which could generate the next task on demand. However, we were experienced with SQS and had designed our architecture around it. Furthermore, it was, in our experience, robust to thousands of machines knocking on it. This does not discount that there could be better methods out there, but this was convenient for us.The two major ways to populate the SQS queue at scale would be a task generating task so a single processor could could enlist hundreds or thousands of others or we could just make our task generating client fast and memory efficient and use a handful of cores for multiprocessing. Keeping things simple and local allows for greater operational flexibility and the addition of a drop-in mutiprocessing execution engine allows for the omission of cloud services for small jobs. Importantly, improved small scale performance doesn't preclude the later development of metageneration facilities.By default, the Python task queue libraries are single threaded and blocking, resulting in upload rates of at most tens of tasks per second. It is possible to do much better by using threads, multiple processes, and by batching requests. TaskQueue has achivied upload rates of over 3000 tasks per second single core, and around 10,000 per second multicore on a single machine. This is sufficient to keep our cluster fed and allows for programmer flexibility as they can populate queues from their local machine using simple scripts.How to Achieve High PerformanceAttaining the quoted upload rates is simple but takes a few tricks to tune the queue. By default, TaskQueue will upload hundreds of tasks per second using its threading model. We'll show via progressive examples how to tune your upload script to get many thousands of tasks per second with near zero latency and memory usage. Note that the examples below usesqs://, but apply tofq://as well. These examples also use the old school style of task instantiation, but you can substitute the new style without consequence.# Listing 1: 10s per second, high memory usage, non-zero latencytasks=[PrintTask(i)foriinrange(1000000)]tq=TaskQueue('sqs://queue-name')fortaskintasks:tq.insert(task)This first example shows how you might use the queue in the most naive fashion. The tasks list takes a long time to compute, uses a lot of memory, and then inserts a single task at a time, failing to exploit the threading model in TaskQueue.Note that this behavior has changed from previous versions where we endorsed the "with" statement where this form was faster, though still problematic.# Listing 2: 100-1000s per second, high memory usage, non-zero latencytasks=[PrintTask(i)foriinrange(1000000)]tq=TaskQueue('sqs://queue-name')tq.insert(tasks)The listing above allows you to use ordinary iterative programming techniques to achieve an upload rate of hundreds per a second without much configuration, a marked improvement over simply using boto nakedly. However, the initial generation of a list of tasks uses a lot of memory and introduces a delay while the list is generated.This form also takes advantage of SQS batch upload which allows for submitting 10 tasks at once. As the overhead for submitting a task lies mainly in HTTP/1.1 TCP/IP connection overhead, batching 10 requests results in nearly a 10x improvement in performance. However, in this case we've created all the tasks up front again in order to batch them correctly which results in the same memory and latency issues as in Listing 1.# Listing 3: 100-1000s per second, low memory usage, near-zero latencytasks=(PrintTask(i)foriinrange(1000000))tq=TaskQueue('sqs://queue-name')tq.insert(tasks,total=1000000)# total necessary for progress bars to workIn Listing 3, we've started using generators instead of lists. Generators are essentially lazy-lists that compute the next list element on demand. Defining a generator is fast and takes constant time, so we are able to begin production of new elements nearly instantly. The elements are produced on demand and consumed instantly, resulting in a small constant memory overhead that can be typically measured in kilobytes to megabytes.As generators do not support thelenoperator, we manually pass in the number of items to display a progress bar.# Listing 4: 100s-1000s per second, low memory usage, near-zero latencyimportgevent.monkeygevent.monkey.patch_all()fromtaskqueueimportTaskQueuetasks=(PrintTask(i)foriinrange(1000000))tq=TaskQueue('sqs://queue-name',green=True)tq.insert(tasks,total=1000000)# total helps the progress barIn Listing 4, we use thegreen=Trueargument to use cooperative threads. Under the hood, TaskQueue relies on Python kernel threads to achieve concurrent IO. However, on systems with mutliple cores, especially those in a virutalized or NUMA context, the OS will tend to distribute the threads fairly evenly between cores leading to high context-switching overhead. Ironically, a more powerful multicore system can lead to lower performance. To remedy this issue, we introduce a user-space cooperative threading model (green threads) using gevent (which depending on your system is uses either libev or libuv for an event loop).This can result in a substantial performance increase on some systems. Typically a single core will be fully utilized with extremely low overhead. However, using cooperative threading with networked IO in Python requires monkey patching the standard library (!!). Refusing to patch the standard library will result in single threaded performance. Thus, using GreenTaskQueue can introduce problems into many larger applications (we've seen problems with multiprocessing and ipython). However, often the task upload script can be isolated from the rest of the system and this allows monkey patching to be safely performed. To give users more control over when they wish to accept the risk of monkey patching, it is not performed automatically and a warning will appear with instructions for amending your program.# Listing 5: 1000s-10000 per second, low memory usage, near zero latency, efficient multiprocessingimportgevent.monkeygevent.monkey.patch_all()fromtaskqueueimportTaskQueuefromconcurrent.futuresimportProcessPoolExecutordefupload(args):start,end=argstasks=(PrintTask(i)foriinrange(start,end))tq=TaskQueue('sqs://queue-name',green=True)tq.insert(tasks,total=(end-start))task_ranges=[(0,250000),(250000,500000),(500000,750000),(750000,1000000)]withProcessPoolExecutor(max_workers=4)aspool:pool.map(upload,task_ranges)In Listing 5, we finally move to multiprocessing to attain the highest speeds. There are three critical pieces of this construction to note.First, we do not use the usualmultiprocessingpackage and instead useconcurrent.futures.ProcessPoolExecutor. If a child process dies inmultiprocessing, the parent process will simply hang (this is by design unfortunately...). Using this alternative package, at least an exception will be thrown.Second, we pass parameters for task generation to the child proceses, not tasks. It is not possible to pass generators from parent to child processes in CPython [1]. It is also inefficient to pass tasks directly as it requires first generating them (as in Listing 1) and then invisibly pickling and unpickling them as they are passed to the child processes. Therefore, we pass only a small number of small picklable objects that are used for constructing a task generator on the other side.Third, as described in the narrative for Listing 5, the GreenTaskQueue has less context-switching overhead than ordinary multithreaded TaskQueue. Using GreenTaskQueue will cause each core to efficiently run independently of the others. At this point, your main bottlenecks will probably be OS/network card related (let us know if they aren't!). Multiprocessing does scale task production, but it's sub-linear in the number of processes. The task upload rate per a process will fall with each additional core added, but each core still adds additional throughput up to some inflection point.# Listing 6: Exchanging Generators for Iteratorsimportgevent.monkeygevent.monkey.patch_all()fromtaskqueueimportTaskQueuefromconcurrent.futuresimportProcessPoolExecutorclassPrintTaskIterator(object):def__init__(self,start,end):self.start=startself.end=enddef__len__(self):returnself.end-self.startdef__iter__(self):foriinrange(self.start,self.end):yieldPrintTask(i)defupload(tsks):tq=TaskQueue('sqs://queue-name',green=True)tq.insert(tsks)tasks=[PrintTaskIterator(0,100),PrintTaskIterator(100,200)]withProcessPoolExecutor(max_workers=2)asexecute:execute.map(upload,tasks)If you insist on wanting to pass generators to your subprocesses, you can use iterators instead. The construction above allows us to write the generator call up front, pass only a few primatives through the pickling process, and transparently call the generator on the other side. We can even support thelen()function which is not available for generators.# Listing 7: Easy Multiprocessingimportgevent.monkeygevent.monkey.patch_all(thread=False)importcopyfromtaskqueueimportTaskQueueclassPrintTaskIterator(object):def__init__(self,start,end):self.start=startself.end=enddef__getitem__(self,slc):itr=copy.deepcopy(self)itr.start=self.start+slc.startitr.end=self.start+slc.stopreturnitrdef__len__(self):returnself.end-self.startdef__iter__(self):foriinrange(self.start,self.end):yieldPrintTask(i)tq=TaskQueue('sqs://queue-name',green=True)tq.insert(PrintTaskIterator(0,200),parallel=2)If you design your iterators such that the slice operator works, TaskQueue can
automatically resection the iterator such that it can be fed to multiple processes. Notably, we don't returnPrintTaskIterator(self.start+slc.start, self.start+slc.stop)because it triggers an endless recursion during pickling. However, the runtime copy implementation above sidesteps this issue. Internally,PrintTaskIterator(0,200)will be turned into[ PrintTaskIterator(0,100), PrintTaskIterator(100,200) ]. We also perform tracking of exceptions raised by child processes in a queue.gevent.monkey.patch_all(thread=False)was necessary to avoid multiprocess hanging.[1] You can't pass generators in CPython butyou can pass iterators. You can pass generators if you use Pypy or Stackless Python.--Made with <3.