Package: md.recipe.celery
Introduction============Buildout recipe for installing Celery for use with Zope's ZCA and configuringit using an ini file.This recipe was originally based on collective.recipe.celery,but differs greatly so that it now has become its own package. This recipedefines a custom loader that can read celery configuration from an ini file.This allows you to generate the configuration by using a template forexample, using collective.recipe.template. The custom loader also bootstrapsthe ZCA automatically when a worker initializes.You can use it in a part like this::[celery]recipe = md.recipe.celeryeggs =${buildout:eggs}celery[redis]celery_conf = etc/celery.inilogging_conf = etc/logging.inizope_conf = etc/zope.confSupported options=================General options---------------eggsA list of additional eggs you want to make available to Celery. Use this toadd additional dependencies such as ``kombu-sqlalchemy`` or the module(s)containing your task definitions.celery_confLocation of the Celery configuration .ini file.logging_confLocation of the file containing the python logging setupzope_confLocation of the zope configuration file.Configuration=============Example of an ini file that could be used by this recipe::[celery]BROKER_URL = sqlite:///celery_broker.dbCELERY_IMPORTS = myapp.tasks,otherapp.tasksCELERY_RESULT_BACKEND = sqlite:///celery_results.dbCELERY_RESULT_SERIALIZER = json[celerybeat:task1]# Execute every hourtask = myapp.tasks.Task1type = crontabschedule = {"minute": 0}[celerybeat:task2]# Execute every 30 secondstask = myapp.tasks.Task2type = timedeltaschedule = {"seconds": 30}[celerybeat:task3]# Execute at midnighttask = otherapp.tasks.Task3type = crontabschedule = {"hour": 0, "minute": 0}The [celery] section defines all the celery options. Every [celerybeat]section defines an individual task.Example usage=============Create a celery ini file holding your celery configuration::>>> write(sample_buildout, 'celery.ini',... """... [celery]... BROKER_URL = sqlite:///celery_broker.db... CELERY_IMPORTS = myapp.tasks,otherapp.tasks... CELERY_RESULT_BACKEND = sqlite:///celery_results.db... CELERY_RESULT_SERIALIZER = json...... [celerybeat:task1]... # Execute every hour... task = myapp.tasks.Task1... type = crontab... schedule = {"minute": 0}...... [celerybeat:task2]... # Execute every 30 seconds... task = myapp.tasks.Task2... type = timedelta... schedule = {"seconds": 30}...... [celerybeat:task3]... # Execute at midnight... task = otherapp.tasks.Task3... type = crontab... schedule = {"hour": 0, "minute": 0}""")This file could be generated by a template such as collective.recipe.template.Next create a logging config::>>> write(sample_buildout, 'logging.ini',... """... [loggers]... keys = root...... [handlers]... keys = console...... [formatters]... keys = generic...... [logger_root]... level = INFO... handlers = console...... [handler_console]... class = StreamHandler... args = (sys.stderr,)... level = NOTSET... formatter = generic...... [formatter_generic]... format = %(asctime)s %(levelname)s [%(name)s] %(message)s""")Then we'll create a buildout that uses the recipe::>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = celery... index = %(server)s/index... find-links = %(server)s...... [celery]... recipe = md.recipe.celery... celery_conf = celery.ini... logging_conf = logging.ini... zope_conf = zope.conf... """% dict(...     server=link_server))Running the buildout gives us::>>> print system(buildout)Installing celery.celery: Creating directory /sample-buildout/parts/celery.celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.Getting distribution for 'celery'.Got celery 2.3.1.Generated script '/sample-buildout/bin/celeryctl'.Generated script '/sample-buildout/bin/celeryd'.<BLANKLINE>Check that we have the celery scripts::>>> ls(sample_buildout, 'bin')-  buildout-  celeryctl-  celerydCheck that we got the celery loaders::>>> ls(sample_buildout, 'parts', 'celery')- loader.py- zopeloader.pyIf we run the celeryd script, it prints out the config data::>>> print(system(join(sample_buildout, 'bin', 'celeryctl')))[('BROKER_URL', 'sqlite:///celery_broker.db'),('CELERYBEAT_SCHEDULE',{'task1': {'schedule': <crontab: 0 * * * * (m/h/d/dM/MY)>,'task': 'myapp.tasks.Task1'},'task2': {'schedule': datetime.timedelta(0, 30),'task': 'myapp.tasks.Task2'},'task3': {'schedule': <crontab: 0 0 * * * (m/h/d/dM/MY)>,'task': 'otherapp.tasks.Task3'}}),('CELERYD_HIJACK_ROOT_LOGGER', False),('CELERY_IMPORTS', ['myapp.tasks', 'otherapp.tasks']),('CELERY_RESULT_BACKEND', 'sqlite:///celery_results.db'),('CELERY_RESULT_SERIALIZER', 'json')]<BLANKLINE>We can include additional eggs using the eggs option::>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = celery... index = %(server)s/index... find-links = %(server)s...... [celery]... recipe = md.recipe.celery... eggs =...     other... """% dict(server=link_server))>>> print system(buildout),Uninstalling celery.Installing celery.celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.Getting distribution for 'other'.Got other 1.0.Generated script '/sample-buildout/bin/celeryctl'.Generated script '/sample-buildout/bin/celeryd'.Generated script '/sample-buildout/bin/distutilsscript'.The recipe should handle updates as well, trigger this by pinning celery toanother version and checking one of the outputted scripts.>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = celery... index = %(server)s/index... find-links = %(server)s...... [versions]... celery = 2.3.0...... [celery]... recipe = md.recipe.celery... eggs =...     other... """% dict(server=link_server))>>> print system(buildout),Updating celery.celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.Getting distribution for 'celery==2.3.0'.Got celery 2.3.0.Generated script '/sample-buildout/bin/celeryctl'.Generated script '/sample-buildout/bin/celeryd'.Generated script '/sample-buildout/bin/distutilsscript'.>>> cat(sample_buildout, 'bin', 'celeryctl')#!/Users/rwendt/Projects/python/bin/python<BLANKLINE>import syssys.path[0:0] = ['/sample-buildout/eggs/celery-2.3.0-py2.7.egg','/sample-buildout/eggs/other-1.0-py2.7.egg','/sample-buildout/parts/celery',]***We can control which scripts are generated using the scripts option.If no value is given, then script generation is disabled::>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = celery... index = %(server)s/index... find-links = %(server)s...... [celery]... recipe = md.recipe.celery... scripts =... """% dict(server=link_server))>>> print system(buildout),Uninstalling celery.Installing celery.celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.>>> ls(sample_buildout, 'bin')-  buildoutLet's create the celeryd script only::>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = celery... index = %(server)s/index... find-links = %(server)s...... [celery]... recipe = md.recipe.celery... scripts =...     celeryd... """% dict(server=link_server))>>> print system(buildout),Uninstalling celery.Installing celery.celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.Generated script '/sample-buildout/bin/celeryd'.>>> ls(sample_buildout, 'bin')-  buildout-  celeryd=======CHANGES=======1.1 (2013-12-19)================- Add missing files to Manifest.1.0 (2013-12-18)================- Initial release. Based loosely on collective.recipe.celery.