Package: pyramid-celery
Getting StartedInclude pyramid_celery either by setting your includes in your .ini,
or by callingconfig.include('pyramid_celery'):pyramid.includes=pyramid_celeryThen you just need to tellpyramid_celerywhat ini file your[celery]section is in:config.configure_celery('development.ini')Then you are free to use celery, for example class based:frompyramid_celeryimportcelery_appasappclassAddTask(app.Task):defrun(self,x,y):printx+yor decorator based:frompyramid_celeryimportcelery_appasapp@app.taskdefadd(x,y):printx+yTo get pyramid settings you may access them inapp.conf['PYRAMID_REGISTRY'].ConfigurationBy defaultpyramid_celeryassumes you want to configure celery via an ini
settings. You can do this by callingconfig.configure_celery(‘development.ini’)but if you are already in themainof your application and want to use the ini
used to configure the app you can do the following:config.configure_celery(global_config['__file__'])If you want to use the standardceleryconfigpython file you can set theuse_celeryconfig = Truelike this:[celery]use_celeryconfig=TrueYou can get more information for celeryconfig.py here:http://celery.readthedocs.io/en/latest/userguide/configuration.htmlAn example ini configuration looks like this:[celery]broker_url=redis://localhost:1337/0imports=app1.tasksapp2.tasks[celery:broker_transport_options]visibility_timeout=18000max_retries=5[celerybeat:task1]task=app1.tasks.Task1type=crontabschedule={"minute": 0}You’ll notice the configuration options that are dictionaries or have
multiple values will be split into their own sections.Scheduled/Periodic TasksTo use celerybeat (periodic tasks) you need to declare 1celerybeatconfig
section per task. The options are:task- The python task you need executed.type- The type of scheduling your configuration uses, options arecrontab,timedelta, andinteger.schedule- The actual schedule for yourtypeof configuration.args- Additional positional arguments.kwargs- Additional keyword arguments.Example configuration for this:[celerybeat:task1]task=app1.tasks.Task1type=crontabschedule={"minute": 0}[celerybeat:task2]task=app1.tasks.Task2type=timedeltaschedule={"seconds": 30}args=[16, 16][celerybeat:task3]task=app2.tasks.Task1type=crontabschedule={"hour": 0, "minute": 0}kwargs={"boom": "shaka"}[celerybeat:task4]task=myapp.tasks.Task4type=integerschedule=30A gotcha you want to watchout for is that the date/time in scheduled tasks
is UTC by default.  If you want to schedule for an exact date/time for your
local timezone you need to settimezone.  Documentation for that
can be found here:http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#time-zonesIf you need to find out what timezones are available you can do the following:frompprintimportpprintfrompytzimportall_timezonespprint(all_timezones)Worker ExecutionThe celerybeat worker will read your configuration and schedule tasks in the
queue to be executed at the time defined.  This means if you are using
celerybeat you will end up running2workers:$celery-Apyramid_celery.celery_appworker--inidevelopment.ini$celery-Apyramid_celery.celery_appbeat--inidevelopment.iniThe first command is the standard worker command that will read messages off
of the queue and run the task. The second command will read the celerybeat
configuration and periodically schedule tasks on the queue.RoutingIf you would like to route a task to a specific queue you can define a route
per task by declaring theirqueueand/orrouting_keyin aceleryroutesection.An example configuration for this:[celeryroute:otherapp.tasks.Task3]queue=slow_tasksrouting_key=turtle[celeryroute:myapp.tasks.Task1]queue=fast_tasksRunning the workerTo run the worker we just use the standard celery command with an additional
argument:celeryworker-Apyramid_celery.celery_app--inidevelopment.iniIf you’ve defined variables in your .ini like %(database_username)s you can use
the–ini-varargument, which is a comma separated list of key value pairs:celeryworker-Apyramid_celery.celery_app--inidevelopment.ini--ini-var=database_username=sontek,database_password=OhYeah!The values inini-varcannot have spaces in them, this will break celery’s
parser.The reason it is a csv instead of using–ini-varmultiple times is because of
a bug in celery itself.  When they fix the bug we will re-work the API. Ticket
is here:https://github.com/celery/celery/pull/2435If you use celerybeat scheduler you need to run with the–beatflag to run
beat and the worker at the same time.celeryworker--beat-Apyramid_celery.celery_app--inidevelopment.iniOr you can launch it separately like this:celerybeat-Apyramid_celery.celery_app--inidevelopment.iniLoggingIf you use the.iniconfiguration (i.e don’t use celeryconfig.py) then the
logging configuration will be loaded from the .ini and will not use the default
celery loggers.You most likely want to add a logging section to your ini for celery as well:[logger_celery]level=INFOhandlers=qualname=celeryand then update your[loggers]section to include it.If you want use the default celery loggers then you can setCELERYD_HIJACK_ROOT_LOGGER=Truein the [celery] section of your .ini.Celery worker processes do not propagate exceptions inside tasks, but swallow them
silently by default. This is related to the behavior of reading asynchronous
task results back. To see if your tasks fail you might need to configurecelery.worker.joblogger to propagate exceptions:# Make sure Celery worker doesn't silently swallow exceptions# See http://stackoverflow.com/a/20719461/315168# https://github.com/celery/celery/issues/2437[logger_celery_worker_job]level=ERRORhandlers=qualname=celery.worker.jobpropagate=1If you want use the default celery loggers then you can setCELERYD_HIJACK_ROOT_LOGGER=Truein the [celery] section of your .iniDemoTo see it all in action check out examples/long_running_with_tm, run
redis-server and then do:$pythonsetup.pydevelop$populate_long_running_with_tmdevelopment.ini$pserve./development.ini$celeryworker-Apyramid_celery.celery_app--inidevelopment.ini4.0.0Drop support for celery 3.0Drop support for Pyramid 2.7Error out if ini is misconfiguredAdd support for pyramid v2Add support for celery v5Drop support for old CELERY config options,
use only the lower case versions.Added support foroptionsin celerybeat.Added support forbroker_transport_options3.0.0Support celery 4.0Properly handle CELERY_ALWAYS_EAGERHandle tuple in ADMINS