Package: pulsar-queue
Badges:Master CI:Downloads:http://pypi.python.org/pypi/pulsar-queueSource:https://github.com/quantmind/pulsar-queueMailing list:google user groupDesign by:QuantmindandLuca SbardellaPlatforms:Linux, OSX, Windows. Python 3.5 and aboveKeywords:server, asynchronous, concurrency, actor, process, queue, tasks, redisAsynchronous server for consuming asynchronous IO tasks, green IO tasks,
blocking IO tasks and long running CPU bound tasks.Fully configurableConsumers poll tasks from distributed message brokers (redis broker implemented)Publish/subscribe for real-time event and logging (redis pub/sub backend)Can schedule tasks when run as a scheduler (--schedule-periodicflag)Build on top ofpulsarandasyncioTL;DRClone the repository:git clone git@github.com:quantmind/pulsar-queue.gitMove to thetests/exampledirectory and run the server:python manage.pyCONTENTSTL;DRFour steps tutorial1 - Create a script2 - Implement Jobs3 - Run the server4 - Queue tasksAPIAPI methodsapi.start()api.on_events(message_type,event_re,callback)api.remove_event_callback(message_type,event_re,callback)api.queue(message,callback=True)api.execute(message)api.consumersTasks APItasks.queue(jobname,**kwargs)tasks.queue_local(jobname,**kwargs)tasks.execute(jobname,*args,**kwargs)tasks.queues()tasks.job_list(jobname=None)The Job classjob.backendjob.default_queuejob.httpjob.loggerjob.max_retriesjob.retry_delayjob.namejob.taskjob.queue(jobname,*args,**kwargs)job.shell(command,**kwargs)The TaskTask StatesEventsConfigurationTasks ConcurrencyASYNC_IOTHREAD_IOCPUBOUNDExtendChangelogLicenseFour steps tutorial1 - Create a scriptA simple python file which runs your application:vim manage.pyfrompq.apiimportPusarQueuetask_paths=['sampletasks.*','pq.jobs']defapp():returnPusarQueue(config=__file__)if__name__=='__main__':app().start()2 - Implement JobsCreate the modules whereJobsare implemented.
It can be a directory containing several submodules.mkdir sampletasks
cd sampletasks
vim mytasks.pyimportasyncioimporttimefrompqimportapi@api.job()defaddition(self,a=0,b=0):returna+b@api.job()asyncdefasynchronous(self,lag=1):start=time.time()awaitasyncio.sleep(lag)returntime.time()-start3 - Run the serverRun the server with two task consumers (pulsar actors).NOTE: Make sure you have Redis server up and running before you start the server.python manage.py -w 24 - Queue tasksLaunch a python shell and play with the api>>>frommanageimportapp>>>api=app().api()>>>task=api.tasks.queue('addition',a=4,b=6)>>>task<TaskFuturependingID=i26ad5c14c5bb422e87b0f7ccbce5ba06>>>>task=task.wait()task.addition<i24ab99ddf2744902a375e039790dcbc4><SUCCESS>>>>task.result10>>>task.status_string'SUCCESS'You can also queue tasks with adelay>>>task=api.tasks.queue('addition',a=4,b=6,callback=False,delay=2).wait()>>>task.status_string'QUEUED'>>>task.time_queued# timestamp>>>task=task.done_callback.wait()>>>task.status_string'SUCCESS'>>>task.time_started-task.time_queued2.00NOTE: Thewaitmethod in a task future can only be used on the shell
or when the event loop is not running. In all other cases one shouldawaitfor the task future in a coroutine.APIThe producer API is obtained from the Task applicationapimethod:frompq.apiimportPusarQueueapi=PusarQueue(...).api()API methodsapi.start()Start listening to events. This method return a coroutine which resolve in the api:api=awaitapi.start()The start method is used when the api is used by application to queue messages/tasks
and listen for events published by distributed consumers.api.on_events(message_type,event_re,callback)Add a callback invoked every time an event matching the regular expressionevent_reoccurs on themessage_typechannel. Thecallbackhas the following signature:defevent_callback(channel,event,message):# event is string, the event matched# message is of type message_typeIf the event is a task event (seeevents) the message is aTaskobject.This method is useful when creating applications which needs to respond to the
queue server events in real time:api.on_events('task', 'queued', callback)
api.on_events('task', 'started', callback)
api.on_events('task', 'done', callback)api.remove_event_callback(message_type,event_re,callback)Remove a previously added event callback. This method is safe.api.queue(message,callback=True)Queue a message in the message queue, equivalent to:api.broker.queue(message,callback)This method returns aMessageFuture, a subclass of asyncioFuturewhich
resolves in amessageobject.
Ifcallbackis True (default) the Future is resolved once the message
is delivered (out of the queue), otherwise is is resolved once the message
is queued (entered the queue).api.execute(message)Execute a message without queueing. This is only supported by messages with
a message consumer which execute them (thetasksconsumer for example).
Ifmessageis aTask, this method is equivalent to:api.tasks.execute(task)This method returns aMessageFuture, a subclass of asyncioFuturewhich
resolve in amessageobject.api.consumersList of consumers registered with the api.Tasks APIThe tasks producer is obtained vua thetasksproperty from the producer API instancetasks=api.tasksThe following methods are available for the tasks producer:tasks.queue(jobname,**kwargs)Queue a task and return aTaskFuturewhich is resolved once the task has finished.
It is possible to obtain a task future resolved when the task has been queued, rather than finished, by passing thecallback=Falseparameter:task=awaittasks.queue(...,callback=False)task.status_string# QUEUEDThekwargsparameters are used as input parameters for theJobcallable with the exception of:callback: discussed abovedelay: delay execution by a given number of secondsqueue: overrides theJob[default_queue](#job-default-queue)[timeout](#job-timeout)meta_params: dictionary of parameters used by theJobcallable to override default values of:
* [max_retries](#job-max-retries)
* [retry_delay](#job-retry-delay)
* [max_concurrency](#job-max-concurrency)tasks.queue_local(jobname,**kwargs)Queue a job in the local task queue. The local task queue is processed by the same server instance. It is equivalent to execute:task=awaittasks.queue(...,queue=tasks.node_name)task.queue# tasks.node_nametasks.execute(jobname,*args,**kwargs)Execute a task immediately, it does not put the task in the task queue.
This method is useful for debugging and testing. It is equivalent to execute:task=awaittasks.queue(...,queue=False)task.queue# Nonetask.status_string# SUCCESStasks.queues()Return the list of queue names the backend is subscribed. This list is not empty when the backend is a task consumer.tasks.job_list(jobname=None)Returns a list ofjob_name,job_descriptiontuples. Thejob_nameis a string which must be used as thejobnameparameter when executing or queing tasks. Thejob_descriptionis a dictionary containing metadata and documentation for the job. Example:jobs=dict(tasks.job_lits())jobs['execute.python']# {#   'type': 'regular',#   'concurrency': 'asyncio',#   'doc_syntax': 'markdown',#   'doc': 'Execute arbitrary python code on a subprocess ... '# }The Job classTheJobclass is how task factories are implemented and added to the
tasks backend registry. When writing a newJobone can either subclass:importasyncioclassAsyncSleep(api.Job):asyncdef__call__(self,lag=1):awaitasyncio.sleep(lag)or use the less verbosejobdecorator:@api.job()asyncdefasyncsleep(self,lag=1):awaitasyncio.sleep(lag)In either cases theselfparameter is an instance of aJobclass and
it has the following useful attributes and methods:job.backendThe tasks backend that is processing thisTaskrunjob.default_queueThe default queue name where tasks for this job are queued. By default it isNonein which case, if aqueueis not given when queueing a task, the first queue
from thequeueslist taken.job.httpBest possible HTTP session handler for the job concurrency mode.job.loggerPython logging handler for this job. The name of this handler
is<app_name>.<job.name>.job.max_retriesOptional positive integer which specify the maximum number of retries when a
task fails or is revoked. If not available failing tasks are not re-queued.
It can be specified as a class attribute or during initialisation from the task
meta parameters.job.retry_delayOptional positive integer which specifies the number of seconds to delay a task
retry.job.nameThe name of this job. Used to queue tasksjob.taskTheTaskinstance associated with this task runjob.queue(jobname,*args,**kwargs)Queue a new job form a task run. It is equivalent to:meta_params={'from_task':self.task.id}self.backend.tasks.queue(...,meta_params=meta_params)job.shell(command,**kwargs)Execute a shell command and returns a coroutine:awaitself.shell("...")The TaskA task contains the metadata information of a job run and it is exchanged between task producers and task consumers via a distributed task queue.Task StatesATaskcan have one of the followingtask.status:QUEUED = 6a task queued but not yet executed.STARTED = 5a task where execution has started.RETRY = 4a task is retrying calculation.REVOKED = 3the task execution has been revoked (or timed-out).FAILURE = 2task execution has finished with failure.SUCCESS = 1task execution has finished with success.FULL_RUN_STATESThe set of states for which aTaskhas run:FAILUREandSUCCESSREADY_STATESThe set of states for which aTaskhas finished:REVOKED,FAILUREandSUCCESSEventsThe task queue broadcast several events during task execution and internal state:task_queued: a newTaskhas been queued, the message is a task instancetask_started: aTaskhas started to be consumed by a task consumer, it is out of the task queuetask_done: aTaskis done, the message is a task in aREADY_STATESConfigurationThere are several parameters you can use to twick the way the task queue works.
In this list the name in bold is the entry point in the config file andcfgdictionary, while, the value between brackets shows the command line entry with default
value.concurrent_tasks(--concurrent-tasks5)The maximum number of concurrent tasks for a given worker in a task consumer server.data_store(--data-storeredis://127.0.0.1:6379/7)Data store used for publishing and subscribing to messages (redis is the
only backend available at the moment)max_requests(--max-requests0)The maximum number of tasks a worker will process before restarting.
A 0 value (the default) means no maximum number, workers will process
all tasks forever.message_broker(--message-broker...)Data store used as distributed task queue. If not provided (default) thedata_storeis used instead. Redis is the
only backend available at the moment.message_serializer(--message-serializerjson)The decoder/encoder for messages and tasks. The default isJSONbutMessage Packis also available ifmsgpackis installed.schedule_periodic(--schedule-periodic)WhenTrue, the task application can schedule periodicJobs.
Usually, only one running server is responsible for
scheduling tasks.task_pool_timeout(--task-pool-timeout2)Timeout in seconds for asynchronously polling tasks from the queues. No need to change this parameter really.workers(--workers4)Number of workers (processes) consuming tasks.Tasks ConcurrencyA task can run in one of fourconcurrencymodes.
If not specified by theJob, the concurrency mode isASYNC_IO.ASYNC_IOThe asynchronous IO mode is associated with tasks which return
an asyncio Future or a coroutine. These tasks run concurrently
in the worker event loop.
An example can be a Job to scrape web pages and create new tasks to process the html@api.job()asyncdefscrape(self,url=None):asserturl,"url is required"request=awaitself.http.get(url)html=request.text()task=self.queue('process.html',html=html,callback=False)returntask.idTHREAD_IOThis concurrency mode is best suited for tasks performingblockingIO operations.
ATHREAD_IOjob runs its tasks in the event loop executor.
You can use this model for most blocking operation unlessLong running CPU boundThe operation does not release the GILExample of tasks suitable for thread IO are IO operations on files.
For example the test suite uses this Job for testingTHREAD_IOconcurrency (check thetests.example.jobs.standardmodule
for the full code):@api.job(concurrency=api.THREAD_IO)defextract_docx(self,input=None,output=None):"""
    Extract text from a docx document
    """importdocxassertinputandoutput,"input and output must be given"document=docx.Document(input)text='\n\n'.join(_docx_text(document))withopen(output,'w')asfp:fp.write(text)return{'thread':threading.get_ident(),'text':len(text)}CPUBOUNDIt assumes the task performs blocking CPU bound operations.
Jobs with this consurrency mode run their tasks on sub-processeses
usingasyncio subprocessmodule.ExtendIt is possible to enhance the task queue application by passing
a customManagerduring initialisation.
For example:frompqimportapiclassManager(api.Manager):asyncdefstore_message(self,message):"""This method is called when a message/task is queued,
        started and finished
        """ifmessage.type=='task':# save this task into a db for exampledefqueues(self):"""List of queue names for Task consumers
        By default it returns the node name and the task_queues
        in the config dictionary.
        """queues=[self.backend.node_name]queues.extend(self.cfg.task_queues)returnqueuestq=PulsarQueue(Manager,...)TheManagerclass is initialised when the backend handler is initialised
(on each consumer and in the scheduler).ChangelogVersions 0.5Versions 0.4Versions 0.3Versions 0.2Versions 0.1LicenseThis software is licensed under the BSD 3-clause License. See the LICENSE
file in the top distribution directory for the full license text. Logo designed by Ralf Holzemer,creative common license.