Package: tasktiger
TaskTigeris a Python task queue using Redis.(Interested in working on projects like this?Closeis looking forgreat engineersto join our team)ContentsFeaturesQuick startConfigurationTask decoratorTask optionsCustom retryingWorkersInspect, requeue and delete tasksPause queue processingRollbar error handlingCleaning Up Error’d TasksRunning The Test SuiteReleasing a New VersionFeaturesPer-task forkTaskTiger forks a subprocess for each task, This comes with several benefits:
Memory leaks caused by tasks are avoided since the subprocess is terminated
when the task is finished. A hard time limit can be set for each task, after
which the task is killed if it hasn’t completed. To ensure performance, any
necessary Python modules can be preloaded in the parent process.Unique queuesTaskTiger has the option to avoid duplicate tasks in the task queue. In some
cases it is desirable to combine multiple similar tasks. For example, imagine
a task that indexes objects (e.g. to make them searchable). If an object is
already present in the task queue and hasn’t been processed yet, a unique
queue will ensure that the indexing task doesn’t have to do duplicate work.
However, if the task is already running while it’s queued, the task will be
executed another time to ensure that the indexing task always picks up the
latest state.Task locksTaskTiger can ensure to never execute more than one instance of tasks with
similar arguments by acquiring a lock. If a task hits a lock, it is requeued
and scheduled for later executions after a configurable interval.Task retryingTaskTiger lets you retry exceptions (all exceptions or a list of specific
ones) and comes with configurable retry intervals (fixed, linear,
exponential, custom).Flexible queuesTasks can be easily queued in separate queues. Workers pick tasks from a
randomly chosen queue and can be configured to only process specific queues,
ensuring that all queues are processed equally. TaskTiger also supports
subqueues which are separated by a period. For example, you can have
per-customer queues in the formprocess_emails.CUSTOMER_IDand start a
worker to processprocess_emailsand any of its subqueues. Since tasks
are picked from a random queue, all customers get equal treatment: If one
customer is queueing many tasks it can’t block other customers’ tasks from
being processed. A maximum queue size can also be enforced.Batch queuesBatch queues can be used to combine multiple queued tasks into one. That way,
your task function can process multiple sets of arguments at the same time,
which can improve performance. The batch size is configurable.Scheduled and periodic tasksTasks can be scheduled for execution at a specific time. Tasks can also be
executed periodically (e.g. every five seconds).Structured loggingTaskTiger supports JSON-style logging via structlog, allowing more
flexibility for tools to analyze the log. For example, you can use TaskTiger
together with Logstash, Elasticsearch, and Kibana.The structlog processortasktiger.logging.tasktiger_processorcan
be used to inject the current task id into all log messages.ReliabilityTaskTiger atomically moves tasks between queue states, and will re-execute
tasks after a timeout if a worker crashes.Error handlingIf an exception occurs during task execution and the task is not set up to be
retried, TaskTiger stores the execution tracebacks in an error queue. The
task can then be retried or deleted manually. TaskTiger can be easily
integrated with error reporting services like Rollbar.Admin interfaceA simple admin interface using flask-admin exists as a separate project
(tasktiger-admin).Quick startIt is easy to get started with TaskTiger.Create a file that contains the task(s).# tasks.pydefmy_task():print('Hello')Queue the task using thedelaymethod.In[1]:importtasktiger,tasksIn[2]:tiger=tasktiger.TaskTiger()In[3]:tiger.delay(tasks.my_task)Run a worker (make sure the task code can be found, e.g. usingPYTHONPATH).%PYTHONPATH=.tasktiger{"timestamp":"2015-08-27T21:00:09.135344Z","queues":null,"pid":69840,"event":"ready","level":"info"}{"task_id":"6fa07a91642363593cddef7a9e0c70ae3480921231710aa7648b467e637baa79","level":"debug","timestamp":"2015-08-27T21:03:56.727051Z","pid":69840,"queue":"default","child_pid":70171,"event":"processing"}Hello{"task_id":"6fa07a91642363593cddef7a9e0c70ae3480921231710aa7648b467e637baa79","level":"debug","timestamp":"2015-08-27T21:03:56.732457Z","pid":69840,"queue":"default","event":"done"}ConfigurationATaskTigerobject keeps track of TaskTiger’s settings and is used to
decorate and queue tasks. The constructor takes the following arguments:connectionRedis connection object. The connection should be initialized withdecode_responses=Trueto avoid encoding problems on Python 3.configDict with config options. Most configuration options don’t need to be
changed, and a full list can be seen withinTaskTiger’s__init__method.Here are a few commonly used options:ALWAYS_EAGERIf set toTrue, all tasks except future tasks (whenis a future
time) will be executed locally by blocking until the task returns. This is
useful for testing purposes.BATCH_QUEUESSet up queues that will be processed in batch, i.e. multiple jobs are taken
out of the queue at the same time and passed as a list to the worker
method. Takes a dict where the key represents the queue name and the value
represents the batch size. Note that the task needs to be declared asbatch=True. Also note that any subqueues will be automatically treated
as batch queues, and the batch value of the most specific subqueue name
takes precedence.ONLY_QUEUESIf set to a non-empty list of queue names, a worker only processes the
given queues (and their subqueues), unless explicit queues are passed to
the command line.setup_structlogIf set to True, sets up structured logging usingstructlogwhen
initializing TaskTiger. This makes writing custom worker scripts easier
since it doesn’t require the user to set upstructlogin advance.Example:importtasktigerfromredisimportRedisconn=Redis(db=1,decode_responses=True)tiger=tasktiger.TaskTiger(connection=conn,config={'BATCH_QUEUES':{# Batch up to 50 tasks that are queued in the my_batch_queue or any# of its subqueues, except for the send_email subqueue which only# processes up to 10 tasks at a time.'my_batch_queue':50,'my_batch_queue.send_email':10,},})Task decoratorTaskTiger provides a task decorator to specify task options. Note that simple
tasks don’t need to be decorated. However, decorating the task allows you to
use an alternative syntax to queue the task, which is compatible with Celery:# tasks.pyimporttasktigertiger=tasktiger.TaskTiger()@tiger.task()defmy_task(name,n=None):print('Hello',name)In[1]:importtasks# The following are equivalent. However, the second syntax can only be used# if the task is decorated.In[2]:tasks.tiger.delay(my_task,args=('John',),kwargs={'n':1})In[3]:tasks.my_task.delay('John',n=1)Task optionsTasks support a variety of options that can be specified either in the task
decorator, or when queueing a task. For the latter, thedelaymethod must
be called on theTaskTigerobject, and any options in the task decorator
are overridden.@tiger.task(queue='myqueue',unique=True)defmy_task():print('Hello')# The task will be queued in "otherqueue", even though the task decorator# says "myqueue".tiger.delay(my_task,queue='otherqueue')When queueing a task, the task needs to be defined in a module other than the
Python file which is being executed. In other words, the task can’t be in the__main__module. TaskTiger will give you back an error otherwise.The following options are supported by bothdelayand the task decorator:queueName of the queue where the task will be queued.hard_timeoutIf the task runs longer than the given number of seconds, it will be
killed and marked as failed.uniqueBoolean to indicate whether the task will only be queued if there is no
similar task with the same function, arguments, and keyword arguments in the
queue. Note that multiple similar tasks may still be executed at the same
time since the task will still be inserted into the queue if another one
is being processed. Requeueing an already scheduled unique task will not
change the time it was originally scheduled to execute at.unique_keyIf set, this impliesunique=Trueand specifies the list of kwargs to use
to construct the unique key. By default, all args and kwargs are serialized
and hashed.lockBoolean to indicate whether to hold a lock while the task is being executed
(for the given args and kwargs). If a task with similar args/kwargs is queued
and tries to acquire the lock, it will be retried later.lock_keyIf set, this implieslock=Trueand specifies the list of kwargs to
use to construct the lock key. By default, all args and kwargs are
serialized and hashed.max_queue_sizeA maximum queue size can be enforced by setting this to an integer value.
TheQueueFullExceptionexception will be raised when queuing a task if
this limit is reached. Tasks in theactive,scheduled, andqueuedstates are counted against this limit.whenTakes either a datetime (for an absolute date) or a timedelta
(relative to now). If given, the task will be scheduled for the given
time.retryBoolean to indicate whether to retry the task when it fails (either because
of an exception or because of a timeout). To restrict the list of failures,
useretry_on. Unlessretry_methodis given, the configuredDEFAULT_RETRY_METHODis used.retry_onIf a list is given, it impliesretry=True. The task will be only retried
on the given exceptions (or its subclasses). To retry the task when a hard
timeout occurs, useJobTimeoutException.retry_methodIf given, impliesretry=True. Pass either:a function that takes the retry number as an argument, or,a tuple(f, args), whereftakes the retry number as the first
argument, followed by the additional args.The function needs to return the desired retry interval in seconds, or raiseStopRetryto stop retrying. The following built-in functions can be
passed for common scenarios and return the appropriate tuple:fixed(delay, max_retries)Returns a method that returns the givendelay(in seconds) or raisesStopRetryif the number of retries exceedsmax_retries.linear(delay, increment, max_retries)Likefixed, but starts off with the givendelayand increments it
by the givenincrementafter every retry.exponential(delay, factor, max_retries)Likefixed, but starts off with the givendelayand multiplies it
by the givenfactorafter every retry.For example, to retry a task 3 times (for a total of 4 executions), and wait
60 seconds between executions, passretry_method=fixed(60, 3).runner_classIf given, a Python class can be specified to influence task running behavior.
The runner class should inherittasktiger.runner.BaseRunnerand implement
the task execution behavior. The default implementation is available intasktiger.runner.DefaultRunner. The following behavior can be achieved:Execute specific code before or after the task is executed (in the forked
child process), or customize the way task functions are called in either
single or batch processing.Note that if you want to execute specific code for all tasks,
you should use theCHILD_CONTEXT_MANAGERSconfiguration option.Control the hard timeout behavior of a task.Execute specific code in the main worker process after a task failed
permanently.This is an advanced feature and the interface and requirements of the runner
class can change in future TaskTiger versions.The following options can be only specified in the task decorator:batchIf set toTrue, the task will receive a list of dicts with args and
kwargs and can process multiple tasks of the same type at once.
Example:[{"args":[1], "kwargs":{}},{"args": [2], "kwargs":{}}]Note that the list will only contain multiple items if the worker
has set upBATCH_QUEUESfor the specific queue (see theConfigurationsection).scheduleIf given, makes a task execute periodically. Pass either:a function that takes the current datetime as an argument.a tuple(f, args), whereftakes the current datetime as the first
argument, followed by the additional args.The schedule function must return the next task execution datetime, orNoneto prevent periodic execution. The function is executed to determine
the initial task execution date when a worker is initialized, and to determine
the next execution date when the task is about to get executed.For most common scenarios, the below mentioned built-in functions can be passed:periodic(seconds=0, minutes=0, hours=0, days=0, weeks=0, start_date=None, end_date=None)Use equal, periodic intervals, starting fromstart_date(defaults to2000-01-01T00:00Z, a Saturday, if not given), ending atend_date(or
never, if not given). For example, to run a task every five minutes
indefinitely, useschedule=periodic(minutes=5). To run a task every
every Sunday at 4am UTC, you could useschedule=periodic(weeks=1, start_date=datetime.datetime(2000, 1, 2, 4)).cron_expr(expr, start_date=None, end_date=None)start_date, to specify the periodic task start date. It defaults to2000-01-01T00:00Z, a Saturday, if not given.end_date, to specify the periodic task end date. The task repeats
forever ifend_dateis not given.
For example, to run a task every hour indefinitely,
useschedule=cron_expr("0* * * *"). To run a task every Sunday at
4am UTC, you could useschedule=cron_expr("04 * * 0").Custom retryingIn some cases the task retry options may not be flexible enough. For example,
you might want to use a different retry method depending on the exception type,
or you might want to like to suppress logging an error if a task fails after
retries. In these cases,RetryExceptioncan be raised within the task
function. The following options are supported:methodSpecify a custom retry method for this retry. If not given, the task’s
default retry method is used, or, if unspecified, the configuredDEFAULT_RETRY_METHOD. Note that the number of retries passed to the
retry method is always the total number of times this method has been
executed, regardless of which retry method was used.original_tracebackIfRetryExceptionis raised from within an except block andoriginal_tracebackis True, the original traceback will be logged (i.e.
the stacktrace at the place where the caught exception was raised). False by
default.log_errorIf set to False and the task fails permanently, a warning will be logged
instead of an error, and the task will be removed from Redis when it
completes. True by default.Example usage:fromtasktiger.exceptionsimportRetryExceptionfromtasktiger.retryimportexponential,fixeddefmy_task():ifnotready():# Retry every minute up to 3 times if we're not ready. An error will# be logged if we're out of retries.raiseRetryException(method=fixed(60,3))try:some_code()exceptNetworkException:# Back off exponentially up to 5 times in case of a network failure.# Log the original traceback (as a warning) and don't log an error if# we still fail after 5 times.raiseRetryException(method=exponential(60,2,5),original_traceback=True,log_error=False)WorkersThetasktigercommand is used on the command line to invoke a worker. To
invoke multiple workers, multiple instances need to be started. This can be
easily done e.g. via Supervisor. The following Supervisor configuration file
can be placed in/etc/supervisor/tasktiger.iniand runs 4 TaskTiger workers
as theubuntuuser. For more information, read Supervisor’s documentation.[program:tasktiger]command=/usr/local/bin/tasktigerprocess_name=%(program_name)s_%(process_num)02dnumprocs=4numprocs_start=0priority=999autostart=trueautorestart=truestartsecs=10startretries=3exitcodes=0,2stopsignal=TERMstopwaitsecs=600killasgroup=falseuser=ubunturedirect_stderr=falsestdout_logfile=/var/log/tasktiger.out.logstdout_logfile_maxbytes=250MBstdout_logfile_backups=10stderr_logfile=/var/log/tasktiger.err.logstderr_logfile_maxbytes=250MBstderr_logfile_backups=10Workers support the following options:-q,--queuesIf specified, only the given queue(s) are processed. Multiple queues can be
separated by comma. Any subqueues of the given queues will be also processed.
For example,-qfirst,secondwill process items fromfirst,second, and subqueues such asfirst.CUSTOMER1,first.CUSTOMER2.-e,--exclude-queuesIf specified, exclude the given queue(s) from processing. Multiple queues can
be separated by comma. Any subqueues of the given queues will also be
excluded unless a more specific queue is specified with the-qoption.
For example,-qemail,email.incoming.CUSTOMER1-eemail.incomingwill
process items from theemailqueue and subqueues likeemail.outgoing.CUSTOMER1oremail.incoming.CUSTOMER1, but notemail.incomingoremail.incoming.CUSTOMER2.-m,--moduleModule(s) to import when launching the worker. This improves task performance
since the module doesn’t have to be reimported every time a task is forked.
Multiple modules can be separated by comma.Another way to preload modules is to set up a custom TaskTiger launch script,
which is described below.-h,--hostRedis server hostname (if different fromlocalhost).-p,--portRedis server port (if different from6379).-a,--passwordRedis server password (if required).-n,--dbRedis server database number (if different from0).-M,--max-workers-per-queueMaximum number of workers that are allowed to process a queue.--store-tracebacks/--no-store-tracebacksStore tracebacks with execution history (config defaults toTrue).In some cases it is convenient to have a custom TaskTiger launch script. For
example, your application may have amanage.pycommand that sets up the
environment and you may want to launch TaskTiger workers using that script. To
do that, you can use therun_worker_with_argsmethod, which launches a
TaskTiger worker and parses any command line arguments. Here is an example:importsysfromtasktigerimportTaskTigertry:command=sys.argv[1]exceptIndexError:command=Noneifcommand=='tasktiger':tiger=TaskTiger(setup_structlog=True)# Strip the "tasktiger" arg when running via manage, so we can run e.g.# ./manage.py tasktiger --helptiger.run_worker_with_args(sys.argv[2:])sys.exit(0)Inspect, requeue and delete tasksTaskTiger provides access to theTaskclass which lets you inspect queues
and perform various actions on tasks.Each queue can have tasks in the following states:queued: Tasks that are queued and waiting to be picked up by the workers.active: Tasks that are currently being processed by the workers.scheduled: Tasks that are scheduled for later execution.error: Tasks that failed with an error.To get a list of all tasks for a given queue and state, useTask.tasks_from_queue. The method gives you back a tuple containing the
total number of tasks in the queue (useful if the tasks are truncated) and a
list of tasks in the queue, latest first. Using theskipandlimitkeyword arguments, you can fetch arbitrary slices of the queue. If you know the
task ID, you can fetch a given task usingTask.from_id. Both methods let
you load tracebacks from failed task executions using theload_executionskeyword argument, which accepts an integer indicating how many executions
should be loaded.Tasks can also be constructed and queued using the regular constructor, which
takes the TaskTiger instance, the function name and the options described in
theTask optionssection. The task can then be queued using itsdelaymethod. Note that thewhenargument needs to be passed to thedelaymethod, if applicable. Unique tasks can be reconstructed using the same
arguments.TheTaskobject has the following properties:id: The task ID.data: The raw data as a dict from Redis.executions: A list of failed task executions (as dicts). An execution
dict contains the processing time intime_startedandtime_failed,
the worker host inhost, the exception name inexception_nameand
the full traceback intraceback.serialized_func,args,kwargs: The serialized function name with
all of its arguments.func: The imported (executable) functionTheTaskobject has the following methods:cancel: Cancel a scheduled task.delay: Queue the task for execution.delete: Remove the task from the error queue.execute: Run the task without queueing it.n_executions: Queries and returns the number of past task executions.retry: Requeue the task from the error queue for execution.update_scheduled_time: Updates a scheduled task’s date to the given date.The current task can be accessed within the task function while it’s being
executed: In case of a non-batch task, thecurrent_taskproperty of theTaskTigerinstance returns the currentTaskinstance. In case of a
batch task thecurrent_tasksproperty must be used which returns a list of
tasks that are currently being processed (in the same order as they were passed
to the task).Example 1: Queueing a unique task and canceling it without a reference to the
original task.fromtasktigerimportTaskTiger,Tasktiger=TaskTiger()# Send an email in five minutes.task=Task(tiger,send_mail,args=['email_id'],unique=True)task.delay(when=datetime.timedelta(minutes=5))# Unique tasks get back a task instance referring to the same task by simply# creating the same task again.task=Task(tiger,send_mail,args=['email_id'],unique=True)task.cancel()Example 2: Inspecting queues and retrying a task by ID.fromtasktigerimportTaskTiger,TaskQUEUE_NAME='default'TASK_STATE='error'TASK_ID='6fa07a91642363593cddef7a9e0c70ae3480921231710aa7648b467e637baa79'tiger=TaskTiger()n_total,tasks=Task.tasks_from_queue(tiger,QUEUE_NAME,TASK_STATE)fortaskintasks:print(task.id,task.func)task=Task.from_id(tiger,QUEUE_NAME,TASK_STATE,TASK_ID)task.retry()Example 3: Accessing the task instances within a batch task function to
determine how many times the currently processing tasks were previously
executed.fromtasktigerimportTaskTigertiger=TaskTiger()@tiger.task(batch=True)defmy_task(args):fortaskintiger.current_tasks:print(task.n_executions())Pause queue processingThe--max-workers-per-queueoption uses queue locks to control the
number of workers that can simultaneously process the same queue. When using
this option a system lock can be placed on a queue which will keep workers
from processing tasks from that queue until it expires. Use theset_queue_system_lock()method of the TaskTiger object to set this lock.Rollbar error handlingTaskTiger comes with Rollbar integration for error handling. When a task errors
out, it can be logged to Rollbar, grouped by queue, task function name and
exception type. To enable logging, initialize rollbar with theStructlogRollbarHandlerprovided in thetasktiger.rollbarmodule. The
handler takes a string as an argument which is used to prefix all the messages
reported to Rollbar. Here is a custom worker launch script:importloggingimportrollbarimportsysfromtasktigerimportTaskTigerfromtasktiger.rollbarimportStructlogRollbarHandlertiger=TaskTiger(setup_structlog=True)rollbar.init(ROLLBAR_API_KEY,APPLICATION_ENVIRONMENT,allow_logging_basic_config=False)rollbar_handler=StructlogRollbarHandler('TaskTiger')rollbar_handler.setLevel(logging.ERROR)tiger.log.addHandler(rollbar_handler)tiger.run_worker_with_args(sys.argv[1:])Cleaning Up Error’d TasksError’d tasks occasionally need to be purged from Redis, soTaskTigerexposes apurge_errored_tasksmethod to help. It might be useful to set
this up as a periodic task as follows:fromtasktigerimportTaskTiger,periodictiger=TaskTiger()@tiger.task(schedule=periodic(hours=1))defpurge_errored_tasks():tiger.purge_errored_tasks(limit=1000,last_execution_before=(datetime.datetime.utcnow()-datetime.timedelta(weeks=12)))Running The Test SuiteTests can be run locally using the provided docker compose file. After installing docker, tests should be runnable with:docker-composerun--rmtasktigerpytestTests can be more granularly run using normal pytest flags. For example:docker-composerun--rmtasktigerpytesttests/test_base.py::TestCaseReleasing a New VersionMake sure the code has been thoroughly reviewed and tested in a realistic production environment.Updatesetup.pyandCHANGELOG.md. Make sure you include any breaking changes.Runpython setup.py sdistandtwine uploaddist/<PACKAGE_TO_UPLOAD>.Push a new tag pointing to the released commit, format:v0.13for example.Mark the tag as a release in GitHub’s UI and include in the description the changelog entry for the version.
An example would be:https://github.com/closeio/tasktiger/releases/tag/v0.13.