Package: pyramid-tasks
Pyramid TasksPyramid and Celery are both fantastic projects that compliment each other well:
Pyramid processes synchronous web requests, while Celery performs asynchronous tasks in the background.
Unfortunately, due to differences in structure and configuration, it's very difficult to integrate the two together.
Configuration, clients, etc. available in Pyramid views may be unavailable in tasks, or may need to be accessed in a different way.
Configuration and functionality may have to be duplicated in order to be shared between Pyramid and Celery.
Pyramid Tasks aims to bridge this gap bycreating a full Pyramid application in the Celery worker and providing a request object to every task.You can use the same configuration for Celery that you do Pyramid, including a Paste-style ini file.
Bringing parity to Pyramid and Celery means you can write code for Pyramid and have the codejust workin Celery.To see Pyramid Tasks in action, check out thesample app.Getting StartedTo use Pyramid Tasks, you should first be familiar withPyramidandCelery.You can install Pyramid Tasks from PyPI:pipinstallpyramid-tasksInclude Pyramid Tasks in your application usingconfig.include, or add it topyramid.includesin your ini file.config.include('pyramid_tasks')Configuring CeleryWhen you import Pyramid Tasks into your application, a new Celery application is created.
All settings prefixed withcelery.are put into Celery's configuration.
As settings from a .ini file are all strings, values are coerced as necessary.
Nested settings are supported by chaining dots, e.g.celery.broker_transport_options.queue_name_prefix.For example, the following simpleceleryconf.py:broker_url='redis://'broker_transport_options={'visibility_timeout':3600,}result_backend='redis://'Is equivalent to the following .ini file:celery.broker_url=redis://celery.broker_transport_options.visibility_timeout=3600celery.result_backend=redis://Running a WorkerIf you're running Pyramid via Paste (i.e. an ini file and possiblypserve),
you can run a Celery worker using the same ini file.celery-Apyramid_tasks--iniconfig.iniThis will create a Pyramid app via the same processpservedoes, allowing you to share configuration between the two environments.You can also create a Celery app usingconfig.make_celery_app(), just like you useconfig.make_wsgi_app().
If you addapp = config.make_celery_app()tocelery.pyin your project's package, you can invokecelery -A myproject workerto boot a worker.To see both methods of running a worker in action, take a look at thesample app.Registering TasksTo register a new task, callconfig.register_taskwith the task function.
You can also use thepyramid_tasks.taskdecorator as long as you run a scan (config.scan()) on the package, just like Pyramid'sview_configdecorator.register_taskand@tasktake the same arguments asCelery.task.For example, a simple Pyramid app with a task might look like the following:frompyramid.configimportConfiguratordefadd(request,x,y):returnx+ywithConfigurator()asconfig:config.register_task(add,name='add')Invoking a TaskOnce a task is registered, you can add it to the work queue usingrequest.defer_task.
This takes the task function or a string of the name of the task as the first argument.
The remaining arguments (positional and keyword) will be passed to the task.
When the task is invoked by a Celery worker, a request object will be created and passed as the first argument.This request object will share the same configuration as requests in the Pyramid application.
This means it will have the same or similar methods, registry, etc.
However, it is nottherequest that invoked the task and properties such asurl,GET, etc. will not be present.
To use these values in your task, pass them in as arguments.Let's take our simple Pyramid app and add a view that invokes the task.frompyramid.configimportConfiguratordefadd_view(context,request):request.defer_task(add,int(request.GET['x']),int(request.GET['y']))return'OK\n'defadd(request,x,y):returnx+ywithConfigurator()asconfig:config.add_route('root','/')config.add_view(add_view,route_name='root')config.register_task(add,name='add')You can also userequest.defer_task_with_optionsto pass options into Celery.
See theCelery docsfor details on what options are available.
For example:defadd_view(context,request):args=int(request.GET['x']),int(request.GET['y'])request.defer_task_with_options(add,args=args,countdown=5)return'OK\n'Getting Task Resultsrequest.defer_taskreturns a CeleryAsyncResultobject.
You can use this object to check if the task has completed (AsyncResult.ready()) and to get the return value of the task (AsyncResult.result).
See the Celery docs for more information.AsyncResultalso has anidproperty.
If you store this property somewhere, such as a client session, you can userequest.get_task_result(id)to return a newAsyncResultobject.pyramid_tm Integrationpyramid_tmis the recommended way of adding transaction management to Pyramid.
For example, thePyramid cookiecutterusespyramid_tmandzope.sqlalchemyto integrate SQLAlchemy into Pyramid.Pyramid Tasks includes built-in support for pyramid_tm.
It can be enabled by includingpyramid_tasks.transactionin your project.
This must be included after Pyramid Tasks, but doesn't need to be included before pyramid_tm.To see Pyramid Tasks, pyramid_tm, and SQLAlchemy in action, check out theSQLAlchemy sample app.Periodic TasksPyramid Tasks supportsCelery Beatfor running periodic tasks.
After registering a task, useconfig.add_periodic_taskto schedule the task.
The arguments mirrorCelery.add_periodic_task:config.add_periodic_task(5.0,# Run every five seconds'mytask',('foo','bar'),# Position arguments passed to task{'fizz':'buzz'},# Keyword arguments passed to task)You can also usecelery.schedules.crontabas the first argument to use crontab-style scheduling.You can run the Beat scheduler the same way you run the Celery worker.celery-Apyramid_tasksbeat--iniconfig.iniTo see Celery Beat in action, check out thebeat sample app.Extending Tasks:  Task DeriversTask Derivers are analogous to Pyramid'sView Derivers.
They allow you to transform the task before registering it, such as wrapping the task in a transaction or adding metric collection.A task deriver is a callable that takes two arguments:  A task function and an info object.
The deriver should return a task callable.
The info object has the following attributes:registry— The registry for the current Pyramid application.package— The package where the configuration statement was found.name— The name of the task.options— The options passed in to the register task action.original_func— The original task function.You can register a new task deriver with theConfigurator.add_task_derivermethod.
The first argument is the task deriver.
The second argument is the name.
If omitted, the name of the task deriver function will be used.
It also optionally takesoverandunderarguments, which work the same as with Pyramid's view deriver.For example, here's a simple task deriver that wraps the task in a database transaction:deftransaction_deriver(task,info):defwrapped(request,*args,**kwargs):withrequest.db:task(request,*args,**kwargs)returnwrappeddefincludeme(config):config.add_task_deriver(transaction_deriver)You can pass in options when registering the task to configure your task derivers.
For example, here's the same transaction task deriver as above, but now will only wrap the task if thein_transactionoption is set.deftransaction_deriver(task,info):defwrapped(request,*args,**kwargs):withrequest.db:task(request,*args,**kwargs)ifinfo.options.get('in_transaction',False):returnwrappedelse:returntaskCelery will accept any keyword arguments passed in, so no configuration is necessary to use your own options.
All options will be set as attributes on the task object.Extending Tasks:  EventsPyramid Task also fires events using Pyramid'sevent system.
Currently the only event ispyramid_task.events.BeforeDeferTask, which will fire when callingdefer_taskordefer_task_with_options.
The event contains the following attributes:request— The current request.task— The task being deferred.args— The arguments being passed to the task.kwargs— The keyword arguments being passed to the task.options— The options being passed intoTask.apply_async.You can modifyoptionsin-place and the changes will be reflected in theapply_asynccall.For example, here's an event subscriber that adds the current user ID to the headers.defadd_headers(event):headers=event.options.setdefault('headers',{})headers.setdefault('user_id',event.request.authenticated_userid)defincludeme(config):config.add_subscriber(add_headers,BeforeDeferTask)The user ID will now be accessible fromrequest.current_task.request.user_id.Fork SafetyCelery by default uses a pre-fork worker model,
meaning the application will be initialized and then forked to launch the desired number of workers.
This can cause issues with some libraries, especially ones utilizing file descriptors such as database connections.
For example,SQLAlchemy requires disposing connections on fork.
You can do this by subscribing to thepyramid_task.events.CeleryWorkerProcessInitevent.``python
config.add_subscriber(lambda _: engine.pool.recreate(), CeleryWorkerProcessInit)The event includes the current [application registry](https://docs.pylonsproject.org/projects/pyramid/en/2.0-branch/api/registry.html) in the `registry` property.

`CeleryWorkerProcessInit` is triggered by Celery's `worker_process_init` signal, so use it in the same situations you would that signal.

## Acknowledgements

Pyramid Tasks is heavily inspired by the code of PyPA's [Warehouse](https://github.com/pypa/warehouse/) project.