Package: redis-limpyd-jobs
redis-limpyd-jobsA queue/jobs system based onredis-limpyd(redisorm (sort of) in python)Where to find it:Github repository:https://github.com/limpyd/redis-limpyd-jobsPypi package:https://pypi.python.org/pypi/redis-limpyd-jobsDocumentation:http://documentup.com/limpyd/redis-limpyd-jobsInstall:Python versions 2.7, and 3.5 to 3.8 are supported (CPython and PyPy).Redis-server versions >= 3 are supported.Redis-py versions >= 3 are supported.Redis-limpyd versions >= 2 are supported.You can still use limpyd-extensions versions < 2 if you need something older than the above requirements.pipinstallredis-limpyd-jobsNote that you actually need theredis-limpyd-extensions(min v1.0)
in addition toredis-limpyd(min v1.2)
(both are automatically installed via pypi)How it worksredis-limpyd-jobsprovides threelimpydmodels (Queue,Job,Error), and aWorkerclass.These models implement the minimum stuff you need to run jobs
asynchronously:Use theJobmodel to store things to doTheQueuemodel will store a list of jobs, with a priority systemTheErrormodel will store all errorsTheWorkerclass is to be used in a process to go through a queue
and run jobsSimple examplefromlimpyd_jobsimportSTATUSES,Queue,Job,Worker# The function to run when a job is called by the workerdefdo_stuff(job,queue):# here do stuff with your jobpass# Create a first job, name 'job:1', in a queue named 'myqueue', with a# priority of 1. The higher the priority, the sooner the job will runjob1=Job.add_job(identifier='job:1',queue_name='myqueue',priority=1)# Add another job in the same queue, with a higher priority, and a different# identifier (if the same was used, no new job would be added, but the# existing job's priority would have been updated)job2=Job.add_job(identifier='job:2',queue_name='myqueue',priority=2)# Create a worker for the queue used previously, asking to call the# "do_stuff" function for each job, and to stop after 2 jobsworker=Worker(queues='myqueue',callback=do_stuff,max_loops=2)# Now really run the jobsworker.run()# Here our jobs are done, our queue is emptyqueue1=Queue.get_queue('myqueue',priority=1)queue2=Queue.get_queue('myqueue',priority=2)# nothing waitingprintqueue1.waiting.lmembers(),queue2.waiting.lmembers()>>[][]# two jobs in success (show PKs of jobs)printqueue1.success.lmembers(),queue2.success.lmembers()>>['limpyd_jobs.models.Job:1','limpyd_jobs.models.Job:2']# Check our jobs statusesprintjob1.status.hget()==STATUSES.SUCCESS>>Trueprintjob2.status.hget()==STATUSES.SUCCESS>>TrueYou notice how it works:Job.add_jobto create a jobWorker()to create a worker, withcallbackargument to set
which function to run for each jobworker.runto launch a worker.Notice that you can run as much workers as you want, even on the same
queue name. Internally, we use theblpopredis command to get jobs
atomically.But you can also run only one worker, having only one queue, doing
different stuff in the callback depending on theidenfitierattribute of the job.Workers are able to catch SIGINT/SIGTERM signals, finishing executing
the current job before exiting. Useful if used, for example, with
supervisord.If you want to store more information in a job, queue or error, or want
to have a different behavior in a worker, it’s easy because you can
create subclasses of everything inlimpyd-jobs, thelimpydmodels or theWorkerclass.ModelsJobA Job stores all needed informations about a task to run.Note: If you want to subclass the Job model to add your own fields,runmethod, or whatever, note that the class must be at the first
level of a python module (ie not in a parent class or function) to work.Job fieldsidentifierA string (InstanceHashField, indexed) to identify the job.When using the (recommended)add_jobclass method, you can’t have
many jobs with the same identifier in a waiting queue. If you create a
new job with an identifier while an other with the same is still in the
same waiting queue, what is done depends on the priority of the two
jobs: - if the new job has a lower (or equal) priority, it’s discarded -
if the new job has a higher priority, the priority of the existing job
is updated to the higher.In both cases theadd_jobclass method returns the existing job,
discarding the new one.A common way of using the identifier is to, at least, store a way to
identify the object on which we want the task to apply: - you can have
one or more queue for a unique task, and store only theidof an
object on theidentifierfield - you can have one or more queue each
doing many tasks, then you may want to store the task too in theidentifierfield: “task:id”Note that by subclassing theJobmodel, you are able to add new
fields to a Job to store the task and other needed parameters, as
arguments (size for a photo to resize, a message to send…)statusA string (InstanceHashField, indexed) to store the actual status of
the job.It’s a single letter but we provide a class to help using it verbosely:STATUSESfromlimpyd_jobsimportSTATUSESprintSTATUSES.SUCCESS>>"s"When a job is created via theadd_jobclass method, its status is
set toSTATUSES.WAITING, orSTATUSES.DELAYEDif it’is delayed by
settingdelayed_until. When it selected by the worker to execute it,
the status passes toSTATUSES.RUNNING. When finished, it’s one ofSTATUSES.SUCCESSorSTATUSES.ERROR. An other available status isSTATUSES.CANCELED, useful if you want to cancel a job without
removing it from its queue.You can also display the full string of a status:printSTATUSES.by_value(my_job.status.hget())>>"SUCCESS"priorityA string (InstanceHashField, indexed, default = 0) to store the
priority of the job.The priority of a job determines in which Queue object it will be
stored. A worker listen for all queues with some names and different
priorities, but respecting the priority (reverse) order: the higher the
priority, the sooner the job will be executed.We choose to use the “`”higher priority is better” way of doing things
to give the possibility to always add a job in a higher priority than
any other ones.Directly updating the priority of a job will not change the queue in
which it’s stored. But when you add a job via the (recommended)add_jobclass method, if a job with the same identifier exists, its
priority will be updated (only if the new one is higher) and the job
will be moved to the higher priority queue.addedA string (InstanceHashField) to store the date and time (a string
representation ofdatetime.utcnow()) of the time the job was added
to its queue.It’s useful in combination of theendfield to calculate the job
duration.startA string (InstanceHashField) to store the date and time (a string
representation ofdatetime.utcnow()) of the time the job was fetched
from the queue, just before the callback is called.It’s useful in combination of theendfield to calculate the job
duration.endA string (InstanceHashField) to store the date and time (a string
representation ofdatetime.utcnow()) of the moment the job was set
as finished or in error, just after the has finished.It’s useful in combination of thestartfield to calculate the job
duration.triesA integer saved as a string (InstanceHashField) to store the number
of times the job was executed. It can be more than one if it was
requeued after an error.delayed_untilThe string representation (InstanceHashField) of adatetimeobject until when the job may be in thedelayedlist (a redis
sorted-set) of the queue.It can be set when callingadd_jobby passing either adelayed_untilargument, which must be adatetime, or adelayed_forargument, which must be a number of seconds (int or
float) or atimedeltaobject. Thedelayed_forargument will be
added to the current time (datetime.utcnow()) to computedelayed_until.If a job is in error after its execution and if the worker has a
positiverequeue_delay_deltaattribute, thedelayed_untilfield
will be set accordingly, useful to retry a erroneous job after a certain
delay.queuedThis field is set to'1'when it’s currently managed by a queue:
waiting, delayed, running. This flag is set when callingenqueue_or_delay, and removed by the worker when the job is
canceled, is finished with success, or finished with error and not
requeued. It’s this field that is checked to test if the same job
already exists whenadd_jobis called.cancel_on_errorYou must be set this field to aTruevalue (don’t forget that Redis
stores Strings, so0will be saved as"0"so it will beTrue… so don’t set it toFalseor0if you want aFalsevalue: yo can let it empty) if you don’t want the job to be
requeued in case of error.Note that if you want to do this for all jobs a a class, you may want to
set toTruethealways_cancel_on_errorattribute of this class.Job attributesqueue_modelWhen adding jobs via theadd_jobmethod, the model defined in this
attribute will be used to get or create a queue. It’s set by default toQueuebut if you want to update it to your own model, you must
subclass theJobmodel too, and update this attribute.queue_nameNoneby default, can be set when overriding theJobclass to
avoid passing thequeue_nameargument to the job’s methods
(especiallyadd_job)Note that if you don’t subclass theJobmodel, you can pass thequeue_modelargument to theadd_jobmethod.always_cancel_on_errorSet this attribute to True if you want all your jobs of this class not
be be requeued in case of error. If you let it to its default value ofFalse, you can still do it job by job by setting their fieldcancel_on_errorto aTruevalue.Job properties and methodsident(property)Theidentproperty is a string representation of the model + the
primary key of the job, saved in queues, allowing the retrieval of the
Job.must_be_cancelled_on_error(property)Themust_be_cancelled_on_errorproperty returns a Boolean indicating
if, in case of error during its execution, the job must NOT be requeued.By default it will beFalse, but there is to way to change this
behavior:setting thealways_cancel_on_errorof your job’s class toTrue.setting thecancel_on_errorfield of your job to aTruevalueduration(property)Thedurationproperty simply returns the time used to compute the
job. The return value is adatetime.timedeltaobject if thestartandendfields are set, orNoneon the other case.run(method)It’s the main method of the job, the only one you must override, to do
some tuff when the job is executed by the worker.The return value of this method will be passed to thejob_successof
the worker, then, if defined, to theon_successmethod of the job.By default aNotImplementederror is raised.Arguments:queue: The queue from which the job was fetched.requeue(method)Therequeuemethod allow a job to be put back in the waiting (or
delayed) queue when its execution failed.Arguments:queue_name=NoneThe queue name in which to save the job. If not
defined, will use the job’s class one. If both are undefined, an
exception is raised.priority=NoneThe new priority of the new job. If not defined,
the job will keep its actual priority.delayed_until=NoneSet this to adatetimeobject to set the
date on which the job will be really requeued. The realdelayed_untilcan also be set by passing thedelayed_forargument.delayed_for=NoneA number of seconds (as a int, float or atimedeltaobject) to wait before the job will be really requeued.
It will compute thedelayed_untilfield of the job.queue_model=NoneThe model to use to store queues. By default,
it’s set toQueue, defined in thequeue_modelattribute of
theJobmodel. If the argument is not set, the attribute will be
used. Be careful to set it as attribute in your subclass, or as
argument inrequeueor the defaultQueuemodel will be used
and jobs won’t be saved in the expected queue model.enqueue_or_delay(method)It’s the method, called inadd_jobandrequeuethat will either
put the job in the waiting or delayed queue, depending ofdelayed_until. If this argument is defined and in the future, the
job is delayed, else it’s simply queued.This method also set thequeuedflag of the job to'1'.Arguments:queue_name=NoneThe queue name in which to save the job. If not
defined, will use the job’s class one. If both are undefined, an
exception is raised.priority=NoneThe new priority of the new job. Use the job’s
actual one if not defined.delayed_until=NoneThe date (must be either adatetimeobject
of the string representation of one) until when the job will remain
in the delayed queue. It will not be processed until this date.prepend=FalseSet toTrueto add the job at the start of the
waiting list, to be the first to be executed (only if not delayed)queue_model=NoneThe model to use to store queues. Seeadd_jobandrequeue.on_started(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called when the job is fetched by the worker and about to be
executed (“waiting” status)Arguments:queue: The queue from which the job was fetched.on_success(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called by the worker when the job’s execution was a success
(it did not raise any exception).Arguments:queue: The queue from which the job was fetched.resultThe data returned by theexecutemethod of the worker,
which call and return the result of therunmethod of the job (or
thecallbackprovided to the worker)on_error(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called by the worker when the job’s execution failed (an
exception was raised)Arguments:queue: The queue from which the job was fetched.exception: The exception that was raised during the execution.traceback: The traceback at the time of the exception, if thesave_tracebacksattribute of the worker was set toTrueon_skipped(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called when the job, just fetched by the worker, could not
be executed because of its status, not “waiting”. Another possible
reason is that the job was canceled during its execution (by settings
its status toSTATUSES.CANCELED)queue: The queue from which the job was fetched.on_requeued(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called by the worker when the job failed and has been
requeued by the worker.queue: The queue from which the job was fetched.on_delayed(ghost method)This method, if defined on your job model (it’s not there by default, ie
“ghost”) is called by the worker when the job was delayed (by settings
its status toSTATUSES.DELAYED) during its execution (note that you
may also want to set thedelayed_untilof the job value to a correct
one datetime (a string represetation of an utc datetime), or the worker
will delay it for 60 seconds).It can also be called if the job’s status was set toSTATUSES.DELAYEDwhile still in thewaitinglist of the queue.queue: The queue from which the job was fetched.Job class methodsadd_jobTheadd_jobclass method is the main (and recommended) way to create
a job. It will check if a job with the same identifier already exists in
a queue (not finished) and if one is found, update its priority (and
move it in the correct queue). If no existing job is found, a new one
will be created and added to a queue.Arguments:identifierThe value for theidentifierfield.queue_name=NoneThe queue name in which to save the job. If not
defined, will use the class one. If both are undefined, an exception
is raised.priority=0The priority of the new job, or the new priority of an
already existing job, if this priority is higher of the existing one.queue_modelThe model to use to store queues. By default, it’s
set toQueue, defined in thequeue_modelattribute of theJobmodel. If the argument is not set, the attribute will be
used. Be careful to set it as attribute in your subclass, or as
argument inadd_jobor the defaultQueuemodel will be used
and jobs won’t be saved in the expected queue model.prepend=FalseBy default, all new jobs are added at the end of
the waiting list (and taken from the start, it’s a fifo list), but
you can force jobs to be added at the beginning of the waiting list
to be the first to be executed, simply by setting theprependargument toTrue. If the job already exists, it will be moved at
the beginning of the list.delayed_until=NoneSet this to adatetimeobject to set the
job to be executed in the future. If defined and in the future, the
job will be added to the delayed list (a redis sorted-set) instead of
the waiting one. The realdelayed_untilcan also be set by
passing thedelayed_forargument.delayed_for=NoneA number of seconds (as a int, float or atimedeltaobject) to wait before adding the job to the waiting
list. It will compute thedelayed_untilfield of the job.If you use a subclass of theJobmodel, you can pass additional
arguments to theadd_jobmethod simply by passing them as named
arguments, they will be save if a new job is created (but not if an
existing job is found in a waiting queue)get_model_reprReturns the string representation of the model, used to compute theidentproperty of a job.get_from_identReturns a job from a string previously got via theidentproperty of
a job.Arguments:identA string including the modele representation of a job and
it’s primary key, as returned by theidentproperty.QueueA Queue stores a list of waiting jobs with a given priority, and keep a
list of successful jobs and ones on error.Queue fieldsnameA string (InstanceHashField, indexed), used by theadd_jobmethod to find the queue in which to store it. Many queues can have the
same names, but different priorities.This name is also used by a worker to find which queues it needs to wait
for.priorityA string (InstanceHashField, indexed, default = 0), to store the
priority of a queue’s jobs. All jobs in a queue are considered having
this priority. It’s why, as said for thepropertyfields of theJobmodel, changing the property of a job doesn’t change its real
property. But adding (via theadd_jobclass method of theJobmodel) a new job with the same identifier for the same queue’s name can
update the job’s priority by moving it to another queue with the correct
priority.As already said, the higher the priority, the sooner the jobs in a queue
will be executed. If a queue has a priority of 2, and another queue of
the same name has a priority of 0, or 1,alljobs in the one with the
priority of 2 will be executed (at least fetched) before the others,
regardless of the number of workers.waitingA list (ListField) to store the primary keys of job in the waiting
status. It’s a fifo list: jobs are appended to the right (viarpush), and fetched from the left (viablpop)When fetched, a job from this list is executed, then pushed in thesuccessorerrorlist, depending if the callback raised an
exception or not. If a job in this waiting list is not in the waiting
status, it will be skipped by the worker.successA list (ListField) to store the primary keys of jobs fetched from
the waiting list and successfully executed.errorA list (ListField) to store the primary keys of jobs fetched from
the waiting list for which the execution failed.delayedA sorted set (SortedSetField) to store delayed jobs, ones having adelayed_untildatetime in the future. The timestamp representation
of thedelayed_untilfield is used as the score for this sorted-set,
to ease the retrieval of jobs that are now ready.Queue attributesTheQueuemodel has no specific attributes.Queue properties and methodsfirst_delayed(property)Returns a tuple representing the first job to be ready in the delayed
queue. It’s a tuple with the job’s pk and the timestamp representation
of it’sdelayed_untilvalue (it’s the score of the sorted_set).Returns None if the delayed queue is empty.first_delayed_time(property)Return the timestamp representation of the first delayed job to be
ready, or None if the delayed queue is empty.delay_job(method)Put a job in the delayed queue.Arguments:jobThe job to delay.delayed_untilAdatetimeobject specifying when the job
should be put back in the waiting queue. It will be converted into a
timestamp used as the score of the delayed list, which is a redis
sorted-set.enqueue_job(method)Put a job in the waiting list.Arguments:jobThe job to enqueue.prepend=FalseSet toTrueto add the job at the start of the
waiting list, to be the first to be executed.requeue_delayed_jobs(method)This method will check for all jobs in the delayed queue that are now
ready to be executed and put them back in the waiting list.This method will return the list of failures, each failure being a tuple
with the value returned by theidentproperty of a job, and the
message of the raised exception causing the failure.Not that the status of the jobs is changed only if their status wasSTATUSES.DELAYED. It allows to cancel a delayed job before.Queue class methodsget_queueTheget_queueclass method is the recommended way to get aQueueobject. Given a name and a priority, it will return the found queue or
create a queue if no matching one exist.Arguments:nameThe name of the queue to get or create.priorityThe priority of the queue to get or create.If you use a subclass of theQueuemodel, you can pass additional
arguments to theget_queuemethod simply by passing them as named
arguments, they will be saved if a new queue is created (but not if an
existing queue is found)get_waiting_keysTheget_waiting_keysclass method returns all the existing (waiting)
queues with the given names, sorted by priority (reverse order: the
highest priorities come first), then by names. The returned value is a
list of redis keys for eachwaitinglists of matching queues. It’s
used internally by the workers as argument to theblpopredis
command.Arguments:namesThe names of the queues to take into accounts (can be a
string if a single name, or a list of strings)count_waiting_jobsThecount_waiting_jobsclass method returns the number of jobs still
waiting for the given queue names, combining all priorities.Arguments:namesThe names of the queues to take into accounts (can be a
string if a single name, or a list of strings)count_delayed_jobsThecount_delayed_jobsclass method returns the number of jobs still
delayed for the given queue names, combining all priorities.Arguments:namesThe names of the queues to take into accounts (can be a
string if a single name, or a list of strings)get_allTheget_allclass method returns a list of queues for the given
names.Arguments:nameThe names of the queues to take into accounts (can be a
string if a single name, or a list of strings)get_all_by_priorityTheget_all_by_priorityclass method returns a list of queues for
the given names, ordered by priorities (the highest priority first),
then names.Arguments:nameThe names of the queues to take into accounts (can be a
string if a single name, or a list of strings)ErrorTheErrormodel is used to store errors from the jobs that are not
successfully executed by a worker.Its main purpose is to be able to filter errors, by queue name, job
model, job identifier, date, exception class name or code. You can use
your own subclass of theErrormodel and then store additional
fields, and filter on them.Error fieldsjob_model_reprA string (InstanceHashField, indexed) to store the string
representation of the job’s model.job_pkA string (InstanceHashField, indexed) to store the primary key of
the job which generated the error.idenfitierA string (InstanceHashField, indexed) to store the identifier of the
job that failed.queue_nameA string (InstanceHashField, indexed) to store the name of the queue
the job was in when it failed.date_timeA string (InstanceHashField, indexed withSimpleDateTimeIndex) to
store the date and time (to the second) of the error (a string representation ofdatetime.utcnow()). This field is indexed so you can filter
errors by date and time (string mode, not by parts of date and time, iedate_time__gt='2017-01-01'), useful to graph errors.dateDEPRECATED: this is replaced bydate_timebut kept for now for compatibilityA string (InstanceHashField, indexed) to store the date (only the
date, not the time) of the error (a string representation ofdatetime.utcnow().date()). This field is indexed so you can filter
errors by date, useful to graph errors.timeDEPRECATED: this is replaced bydate_timebut kept for now for compatibilityA string (InstanceHashField) to store the time (only the time, not
the date) of the error (a string representation ofdatetime.utcnow().time()).typeA string (InstanceHashField, indexed) to store the type of error.
It’s the class’ name of the originally raised exception.codeA string (InstanceHashField, indexed) to store the value of thecodeattribute of the originally raised exception. Nothing is stored
here if there is no such attribute.messageA string (InstanceHashField) to store the string representation of
the originally raised exception.tracebackA string (InstanceHashField) to store the string representation of
the traceback of the originally raised exception (the worker may not
have filled it)Error properties and methodsdatetimeThis property returns adatetimeobject based on the content of thedate_timefield of anErrorobject.Error class methodsadd_errorTheadd_errorclass method is the main (and recommended) way to add
an entry on theErrormodel, by accepting simple arguments that will
be break down (jobbecomesidentifierandjob_pk,whenbecomesdateandtime,errorbecomescodeandmessage)Arguments:queue_nameThe name of the queue the job came from.jobThe job which generated the error, from which we’ll extractjob_pkandidentifiererrorAn exception from which we’ll extract the code and the
message.when=NoneAdatetimeobject from which we’ll extract the date
and time.If not filled,datetime.utcnow()will be used.trace=NoneThe traceback, stringyfied, to store.If you use a subclass of theErrormodel, you can pass additional
arguments to theadd_errormethod simply by passing them as named
arguments, they will be save in the object to be created.collection_for_jobThecollection_for_jobis a helper to retrieve the errors assiated
with a given job, more precisely for all the instances of this job with
the same identifier.The result is alimpydcollection, to you can usefilter,instances… on it.Arguments:jobThe job for which we want errorsThe worker(s)The Worker classTheWorkerclass does all the logic, working withQueueandJobmodels.The main behavior is: - reading queue keys for the given names - waiting
for a job available in the queues - executing the job - manage success
or error - exit after a defined number of jobs or a maximum duration (if
defined), or when aSIGINT/SIGTERMsignal is caughtThe class is split in many short methods so that you can subclass it to
change/add/remove whatever you want.Constructor arguments and worker’s attributesEach of the following worker’s attributes can be set by an argument in
the constructor, using the exact same name. It’s why the two are
described here together.queuesNames of the queues to work with. It can be a list/tuple of strings, or
a string with names separated by a comma (no spaces), or without comma
for a single queue.Note that all queues must be from the samequeue_model.Default toNone, but if not set and not defined in a subclass, will
raise anLimpydJobsException.queue_modelThe model to use for queues. By default it’s theQueuemodel
included inlimpyd_jobs, but you can use a subclass of the default
model to add fields, methods…error_modelThe model to use for saving errors. By default it’s theErrormodel
included inlimpyd_jobs, but you can use a subclass of the default
model to add fields, methods…logger_namelimpyd_jobsuses the pythonloggingmodule, so this is the name
to use for the logger created for the worker. The default value isLOGGER_NAME, withLOGGER_NAMEdefined inlimpyd_jobs.workerswith a value of “limpyd-jobs”.logger_levelIt’s the level set for the logger created with the name defined inlogger_name, default tologging.INFO.save_errorsA boolean, default toTrue, to indicate if we have to save errors in
theErrormodel (or the one defined inerror_model) when the
execution of the job is not successful.save_tracebacksA boolean, default toTrue, to indicate if we have to save the
tracebacks of exceptions in theErrormodel (or the one defined inerror_model) when the execution of the job is not successful (and
only ifsave_errorsisTrue)max_loopsThe max number of loops (fetching + executing a job) to do in the worker
lifetime, default to 1000. Note that after this number of loop, the
worker ends (therunmethod cannot be executed again)The aim is to avoid memory leaks become too important.max_durationIf defined, the worker will end when itsrunmethod was called for
at least this number of seconds. By default it’s set toNone, saying
there is no maximum duration.terminate_gracefullyTo avoid interrupting the execution of a job, ifterminate_gracefullyis set toTrue(the default), theSIGINTandSIGTERMsignals are caught, asking the worker to exit
when the current jog is done.callbackThe callback is the function to run when a job is fetched. By default
it’s theexecutemethod of the worker (which calls therunmethod of jobs, which, if not overridden, raises aNotImplementederror) , but you can pass any function that accept a job and a queue as
argument.Using the queue’s name, and the job’s identifier+model (viajob.ident), you can manage many actions depending on the queue if
needed.If this callback (or theexecutemethod) raises an exception, the
job is considered in error. In the other case, it’s considered
successful and the return value is passed to thejob_successmethod,
to let you do what you want with it.timeoutThe timeout is used as parameter to theblpopredis command we use
to fetch jobs from waiting lists. It’s 30 seconds by default but you can
change it to any positive number (in seconds). You can set it to0if you don’t want any timeout be applied to theblpopcommand.It’s better to always set a timeout, to reenter the main loop and call
themust_stopmethod to see if the worker must exit. Note that the
number of loops is not updated in the case of the timeout occurred, so a
littletimeoutwon’t alter the number of loops defined bymax_loops.fetch_priorities_delayThefetch_priorities_delayis the delay between two fetches of the
list of priorities for the current worker.If a job was added with a priority that did not exist when the worker
run was started, it will not be taken into account until this delay
expires.Note that if this delay is, say, 5 seconds (it’s 25 by default), and thetimeoutparameter is 30, you may wait 30 seconds before the new
priority fetch because if there is no jobs in the priority queues
actually managed by the worker, the time is in the redis hands.fetch_delayed_delayThefetch_delayed_delayis the delay between two fetches of the
delayed jobs that are now ready in the queues managed by the worker.Note that if this delay is, say, 5 seconds (it’s 25 by default), and thetimeoutparameter is 30, you may wait 30 seconds before the new
delayed fetch because if there is no jobs in the priority queues
actually managed by the worker, the time is in the redis hands.requeue_timesIt’s the number of times a job will be requeued when its execution
results in a failure. It will then be put back in the same queue.This attribute is 0 by default so by default a job won’t be requeued.requeue_priority_deltaThis number will be added to the current priority of the job that will
be requeued. By default it’s set to -1 to decrease the priority at each
requeue.requeue_delay_deltaIt’s a number of seconds to wait before adding back an erroneous job in
the waiting queue, set by default to 30: when a job failed to execute,
it’s put in the delayed queue for 30 seconds then it’ll be put back in
the waiting queue (depending on thefetch_delayed_delayattribute)Other worker’s attributesIn case on subclassing, you can need these attributes, created and
defined during the use of the worker:keysA list of keys of queues waiting lists, which are listened by the worker
for new jobs. Filled by theupdate_keysmethod.statusThe current status of the worker.Noneby default until therunmethod is called, after what it’s set to"starting"while getting
for an available queue. Then it’s set to"waiting"while the worker
waits for new jobs. When a job is fetched, the status is set to"running". And finally, when the loop is over, it’s set to"terminated".If the status is notNone, therunmethod cannot be called.loggerThe logger (from theloggingpython module) defined by theset_loggermethod.num_loopsThe number of loops done by the worker, incremented each time a job is
fetched from a waiting list, even if the job is skipped (bad status…),
or in error. When this number equals themax_loopsattribute, the
worker ends.end_forcedWhenTrue, ask for the worker to terminate itself after executing
the current job. It can be set toTruemanually, or when a
SIGINT/SIGTERM signal is caught.end_signal_caughtThis boolean is set toTruewhen a SIGINT/SIGTERM is caught (only if
theterminate_gracefullyisTrue)start_dateNoneby default, set todatetime.utcnow()when therunmethod starts.end_dateNoneby default, set todatetime.utcnow()when therunmethod ends.wanted_end_dateNone by default, it’s computed to know when the worker must stop based
on thestart_dateandmax_duration. It will always beNoneif nomax_durationis defined.connectionIt’s a property, not an attribute, to get the current connection to the
redis server.parametersIt’s a tuple holding all parameters accepted by the worker’s constructorparameters=('queues','callback','queue_model','error_model','logger_name','logger_level','save_errors','save_tracebacks','max_loops','max_duration','terminate_gracefuly','timeout','fetch_priorities_delay','fetch_delayed_delay','requeue_times','requeue_priority_delta','requeue_delay_delta')Worker’s methodsAs said before, theWorkerclass in spit in many little methods, to
ease subclassing. Here is the list of public methods:__init__Signature:def__init__(self,queues=None,**kwargs):Returns nothing.It’s the constructor (you guessed it ;) ) of theWorkerclass,
expecting all arguments (defined inparameters) that can also be
defined as class attributes.It validates these arguments, prepares the logging and initializes other
attributes.You can override it to add, validate, initialize other arguments or
attributes.handle_end_signalSignature:defhandle_end_signal(self):Returns nothing.It’s called in the constructor ifterminate_gracefullyisTrue.
It plugs the SIGINT and SIGTERM signal to thecatch_end_signalmethod.You can override it to catch more signals or do some checked before
plugging them to thecatch_end_signalmethod.stop_handling_end_signalSignature:defstop_handling_end_signal(self):Returns nothing.It’s called at the end of therunmethod, as we don’t need to catch
the SIGINT and SIGTERM signals anymore. It’s useful when launching a
worker in a python shell to finally let the shell handle these signals.
Useless in a script because the script is finished when therunmethod exits.set_loggerSignature:defset_logger(self):Returns nothing.It’s called in the constructor to initialize the logger, usinglogger_nameandlogger_level, saving it inself.logger.must_stopSignature:defmust_stop(self):Returns boolean.It’s called on the main loop, to exit it on some conditions: an end
signal was caught, themax_loopsnumber was reached, orend_forcedwas set toTrue.wait_for_jobSignature:defwait_for_job(self):Returns a tuple with a queue and a jobThis method is called during the loop, to wait for an available job in
the waiting lists. When one job is fetched, returns the queue (an
instance of the model defined byqueue_model) on which the job was
found, and the job itself.get_jobSignature:defget_job(self,job_ident):Returns a job.Called duringwait_for_jobto get a real job object based on the
job’sident(model + pk) fetched from the waiting lists.get_queueSignature:defget_queue(self,queue_redis_key):Returns a Queue.Called duringwait_for_jobto get a real queue object (an instance
of the model defined byqueue_model) based on the key returned by
redis telling us in which list the job was found. This key is not the
primary key of the queue, but the redis key of it’s waiting field.catch_end_signalSignature:defcatch_end_signal(self,signum,frame):Returns nothing.It’s called when a SIGINT/SIGTERM signal is caught. It’s simply setend_signal_caughtandend_forcedtoTrue, to tell the worker
to terminate as soon as possible.executeSignature:defexecute(self,job,queue):Returns nothing by default.This method is called if nocallbackargument is provided when
initiating the worker and call therunmethod of the job, which
raises aNotImplementedErrorby default.If the execution is successful, no return value is attended, but if any,
it will be passed to thejob_successmethod. And if an error
occurred, an exception must be raised, which will be passed to thejob_errormethod.update_keysSignature:defupdate_keys(self):Returns nothing.Calling this method updates the internalkeysattributes, which
contains redis keys of the waiting lists of all queues listened by the
worker.It’s actually called at the beginning of therunmethod, and at
intervals depending onfetch_priorities_delay. Note that if a queue
with a specific priority doesn’t exist when this method is called, but
later, by adding a job withadd_job, the worker will ignore it
unless thisupdate_keysmethod was called again (programmatically or
by waiting at leastfetch_priorities_delayseconds)runSignature:defrun(self):Returns nothing.It’s the main method of the worker, with all the logic: while we don’t
have to stop (result of themust_stopmethod), fetch a job from
redis, and if this job is really in waiting state, execute it, and do
something depending of the status of the execution (success, error…).In addition to the methods that do real stuff (update_keys,wait_for_job), some other methods are called during the execution:run_started,run_ended, about the run, andjob_skipped,job_started,job_successandjob_errorabout jobs. You can
override these methods in subclasses to adapt the behavior depending on
your needs.run_startedSignature:defrun_started(self):Returns nothing.This method is called in therunmethod after the keys are computed
usingupdate_keys, just before starting the loop. By default it does
nothing but a log.info.run_endedSignature:defrun_ended(self):Returns nothing.This method is called just before exiting therunmethod. By default
it does nothing but a log.info.job_skippedSignature:defjob_skipped(self,job,queue):Returns nothing.When a job is fetched in therunmethod, its status is checked. If
it’s notSTATUSES.WAITING, thisjob_skippedmethod is called,
with two main arguments: the job and the queue in which it was found.This method is also called when the job is canceled during its execution
(ie if, when the execution is done, the job’s status isSTATUSES.CANCELED).This method remove thequeuedflag of the job, logs the message
returned by thejob_skipped_messagemethod, then call, if defined,
theon_skippedmethod of the job.job_skipped_messageSignature:defjob_skipped_message(self,job,queue):Returns a string to be logged injob_skipped.job_startedSignature:defjob_started(self,job,queue):Returns nothing.When the job is fetched and its status verified (it must beSTATUSES.WAITING), thejob_startedmethod is called, just before
the callback (or theexecutemethod if nocallbackis defined),
with the job and the queue in which it was found.This method updates thestartandstatusfields of the job, then
log the message returned byjob_started_messageand finally call, if
defined, theon_startedmethod of the job.job_started_messageSignature:defjob_started_message(self,job,queue):Returns a string to be logged injob_started.job_successSignature:defjob_success(self,job,queue,job_result):Returns nothing.When the callback (or theexecutemethod) is finished, without
having raised any exception, the job is considered successful, and thejob_successmethod is called, with the job and the queue in which it
was found, and the return value of the callback method.Note that this method is not called, and so the job not considered a
“success” if, when the execution is done, the status of the job is
eitherSTATUS.CANCELEDorSTATUS.DELAYED. In these cases, the
methodsjob_skippedandjob_delayedare called respectively.This method remove thequeuedflag of the job, updates itsendandstatusfields, moves the job into thesuccesslist of the
queue, then log the message returned byjob_success_messageand
finally call, if defined, theon_successmethod of the job.job_success_messageSignature:defjob_success_message(self,job,queue,job_result):Returns a string to be logged injob_success.job_delayedSignature:defjob_delayed(self,job,queue):Returns nothing.When the callback (or theexecutemethod) is finished, without
having raised an exception, and the status of the job at this moment isSTATUSES.DELAYED, the job is not successful but not in error: it
will be delayed.Another way to have this method called if its a job is in thewaitingqueue but its status was set toSTATUSES.DELAYED. In
this cas, the job is not executed, but delayed by calling this method.This method check if the job has adelayed_untilvalue, and if not,
or if an invalid one, it is set to 60 seconds in the future. You may
want to explicitly set this value, or at least clear the field because
if the job was initially delayed, the value may be set, but in the past,
and the job will be delayed to this date, so, not delayed but just
queued.With this value, the methodenqueue_or_delayof the queue is called,
to really delay the job.Then, log the message returned byjob_delayed_messageand finally
call, if defined, theon_delayedmethod of the job.job_delayed_messageSignature:defjob_delayed_message(self,job,queue):Returns a string to be logged injob_delayed.job_errorSignature:defjob_error(self,job,queue,exception,trace=None):Returns nothing.When the callback (or theexecutemethod) is terminated by raising
an exception, thejob_errormethod is called, with the job and the
queue in which it was found, and the raised exception and, ifsave_tracebacksisTrue, the traceback.This method remove thequeuedflag of the job if it is no to be
requeued, updates itsendandstatusfields, moves the job into
theerrorlist of the queue, adds a new error object (ifsave_errorsisTrue), then log the message returned byjob_error_messageand call theon_errormethod of the job is
called, if defined.And finally, if themust_be_cancelled_on_errorproperty of the job
is False, and therequeue_timesworker attribute allows it
(considering thetriesattribute of the job, too), therequeue_jobmethod is called.job_error_messageSignature:defjob_error_message(self,job,queue,to_be_requeued_exception,trace=None):Returns a string to be logged injob_error.job_requeue_messageSignature:defjob_requeue_message(self,job,queue):Returns a string to be logged injob_errorwhen the job was
requeued.additional_error_fieldsSignature:defadditional_error_fields(self,job,queue,exception,trace=None):Returns a dictionary of fields to add to the error object, empty by
default.This method is called byjob_errorto let you define a dictionary of
fields/values to add to the error object which will be created, if you
use a subclass of theErrormodel, defined inerror_model.To pass these additional fields to the error object, you have to
override this method in your own subclass.requeue_jobdefrequeue_job(self,job,queue,priority,delayed_for=None):Returns nothing.This method is called to requeue the job when its execution failed, and
will call therequeuemethod of the job, then itsrequeuedone,
and finally will log the message returned byjob_requeue_message.idIt’s a property returning a string identifying the current worker, used
in logging to distinct log entries for each worker.elapsedIt’s a property returning, when running the time elapsed since when therunstarted. When therunmethod ends, it’s the time betweenstart_dateandend_date.If therunmethod is not called, it will be set toNone.logSignature:deflog(self,message,level='info'):Returns nothing.logis a simple wrapper aroundself.logger, which automatically
add theidof the worker at the beginning. It can accepts alevelargument which isinfoby default.set_statusSignature:defset_status(self,status):Returns nothing.set_statussimply update the worker’sstatusfield.count_waiting_jobsSignature:defcount_waiting_jobs(self):Returns the number of jobs in waiting state that can be run by this
worker.count_delayed_jobsSignature:defcount_delayed_jobs(self):Returns the number of jobs in the delayed queues managed by this worker.The worker.py scriptTo help usinglimpyd_jobs, an executable python script is provided:scripts/worker.py(usable aslimpyd-jobs-worker, in your path,
when installed from the package)This script is highly configurable to help you launching workers without
having to write a script or customize the one included.With this script you don’t have to write a custom worker too, because
all arguments attended by a worker can be passed as arguments to the
script.The script is based on aWorkerConfigclass defined inlimpyd_jobs.workers, that you can customize by subclassing it, and
you can tell the script to use your class instead of the default one.You can even pass one or many python paths to add tosys.path.This script is designed to ease you as much as possible.Instead of explaining all arguments, see below the result of the--helpcommand for this script:$ limpyd-jobs-worker  --help
Usage: worker.py [options]

Run a worker using redis-limpyd-jobs

Options:
  --pythonpath=PYTHONPATH
                        A directory to add to the Python path, e.g.
                        --pythonpath=/my/module
  --worker-config=WORKER_CONFIG
                        The worker config class to use, e.g. --worker-
                        config=my.module.MyWorkerConfig, default to
                        limpyd_jobs.workers.WorkerConfig
  --print-options       Print options used by the worker, e.g. --print-options
  --dry-run             Won't execute any job, just starts the worker and
                        finish it immediatly, e.g. --dry-run
  --queues=QUEUES       Name of the Queues to handle, comma separated e.g.
                        --queues=queue1,queue2
  --queue-model=QUEUE_MODEL
                        Name of the Queue model to use, e.g. --queue-
                        model=my.module.QueueModel
  --error-model=ERROR_MODEL
                        Name of the Error model to use, e.g. --queue-
                        model=my.module.ErrorModel
  --worker-class=WORKER_CLASS
                        Name of the Worker class to use, e.g. --worker-
                        class=my.module.WorkerClass
  --callback=CALLBACK   The callback to call for each job, e.g. --worker-
                        class=my.module.callback
  --logger-name=LOGGER_NAME
                        The base name to use for logging, e.g. --logger-base-
                        name="limpyd-jobs.%s"
  --logger-level=LOGGER_LEVEL
                        The level to use for logging, e.g. --worker-class=ERROR
  --save-errors         Save job errors in the Error model, e.g. --save-errors
  --no-save-errors      Do not save job errors in the Error model, e.g. --no-
                        save-errors
  --save-tracebacks     Save exception tracebacks on job error in the Error
                        model, e.g. --save-tracebacks
  --no-save-tracebacks  Do not save exception tracebacks on job error in the
                        Error model, e.g. --no-save-tracebacks
  --max-loops=MAX_LOOPS
                        Max number of jobs to run, e.g. --max-loops=100
  --max-duration=MAX_DURATION
                        Max duration of the worker, in seconds (None by
                        default), e.g. --max-duration=3600
  --terminate-gracefuly
                        Intercept SIGTERM and SIGINT signals to stop
                        gracefuly, e.g. --terminate-gracefuly
  --no-terminate-gracefuly
                        Do NOT intercept SIGTERM and SIGINT signals, so don't
                        stop gracefuly, e.g. --no-terminate-gracefuly
  --timeout=TIMEOUT     Max delay (seconds) to wait for a redis BLPOP call (0
                        for no timeout), e.g. --timeout=30
  --fetch-priorities-delay=FETCH_PRIORITIES_DELAY
                        Min delay (seconds) to wait before fetching new
                        priority queues, e.g. --fetch-priorities-delay=20
  --fetch-delayed-delay=FETCH_DELAYED_DELAY
                        Min delay (seconds) to wait before updating delayed
                        jobs, e.g. --fetch-delayed-delay=20
  --requeue-times=REQUEUE_TIMES
                        Number of time to requeue a failing job (default to
                        0), e.g. --requeue-times=5
  --requeue-priority-delta=REQUEUE_PRIORITY_DELTA
                        Delta to add to the actual priority of a failing job
                        to be requeued (default to -1, ie one level lower),
                        e.g. --requeue-priority-delta=-2
  --requeue-delay-delta=REQUEUE_DELAY_DELTA
                        How much time (seconds) to delay a job to be requeued
                        (default to 30), e.g. --requeue-delay-delta=15
  --database=DATABASE   Redis database to use (host:port:db), e.g.
                        --database=localhost:6379:15
  --no-title            Do not update the title of the worker's process, e.g.
                        --no-title
  --version             show program's version number and exit
  -h, --help            show this help message and exitExcept for--pythonpath,--worker-config,--print-options,--dry-run,--worker-classand--no-title, all options will be passed to the worker.So, if you use the default models, the default worker with its default
options, and to launch a worker to work on the queue “queue-name”, all
you need to do is:limpyd-jobs-worker--queues=queue-name--callback=python.path.to.callbackWe use thesetproctitlemodule to display useful informations in the
process name, to have stuff like this:limpyd-jobs-worker#1566090 [init] queues=foo,bar
limpyd-jobs-worker#1566090 [starting] queues=foo,bar loop=0/1000 waiting=10 delayed=0
limpyd-jobs-worker#1566090 [running] queues=foo,bar loop=1/1000 waiting=9 delayed=2 duration=0:00:15
limpyd-jobs-worker#1566090 [terminated] queues=foo,bar loop=10/1000 waiting=0 delayed=0 duration=0:12:27You can disable it by passing the--no-titleargument.Note that if no logging handler is set for thelogger-name, aStreamHandlerformatter will be automatically added by the script,
given logs like:[19122] 2013-10-02 00:51:24,158 (limpyd-jobs) WARNING  [038480] [test|job:1] job skipped (current status: SUCCESS)(the format used is"[%(process)d]%(asctime)s(%(name)s)%(levelname)-8s%(message)s")Executing code before loading worker classSometimes you may want to do some initialization work before even
loading the Worker class, for example, using django, to adddjango.setup()For this, simple override theWorkerConfigclass:importdjangofromlimpyd_jobs.workersimportWorkerConfigclassMyWorkerConfig(WorkerConfig):def__init__(self,argv=None):django.setup()super(MyWorkerConfig,self).__init__(argv)And pass the python path to this class using the--worker-configoption to thelimpyd-jobs-workerscript.TestsTheredis-limpyd-jobspackage is fully tested (coverage: 100%).To run the tests, which are not installed via thesetup.pyfile, you
can do:$ python run_tests.py
[...]
Ran 136 tests in 19.353s

OKOr if you havenosetestsinstalled:$ nosetests
[...]
Ran 136 tests in 20.471s

OKThenosetestsconfiguration is provided in thesetup.cfgfile
and include the coverage, ifnose-covis installed.Final wordsyou can see a full example inexample.py(in the source, not in
the installed package)to uselimpyd_jobsmodels on your own redis database instead of
the default one (localhost:6379:db=0), simply use theuse_databasemethod of the main model:fromlimpyd.contrib.databaseimportPipelineDatabasefromlimpyd_jobs.modelsimportBaseJobsModeldatabase=PipelineDatabase(host='localhost',port=6379,db=15)BaseJobsModel.use_database(database)or simply change the connection settings:fromlimpyd_jobs.modelsimportBaseJobsModelBaseJobsModel.database.connect(host='localhost',port=6379,db=15)The end.