Package: cadasta-workertoolbox
A collection of helpers to assist in quickly building asynchronous
workers for the Cadasta system.Librarycadasta.workertoolbox.conf.ConfigTheConfigclass was built to simplify configuring Celery settings,
helping to ensure that all workers adhere to the architecture
requirements of the Cadasta asynchronous system. It essentially offers a
diff between Celery’s default configuration and the configuration
required by our system. It is the aim of the class to not require much
customization on the part of the developer, however some customization
may be needed when altering configuration between environments (e.g. if
dev settings vary greatly from prod settings).AnyCelery
settingmay be submitted via keyword argument or via environment variable.
Arguments submitted via keyword argument are expected to comply with
Celery’s newer lowercase settings rather than their older uppercase
counterparts. Arguments provided by environment variable should be
uppercase and be prepended with the prefixCELERY_(e.g. to set thetask_track_startedvalue, an environment variable ofCELERY_TASK_TRACK_STARTEDshould be set). The prefix can be
customized with a providedENV_PREFIXkeyword argument orCELERY_ENV_PREFIXenvironment variable. If both a keyword argument
and environment variable are provided for a setting, the keyword
argument takes precedence. Settings with non-string defaults will have
the environment variable values run through`ast.literal_eval<https://docs.python.org/3/library/ast.html#ast.literal_eval>`__,
supporting Python native types likeboolortuple. Only
lowercase settings are shown when callingrepron theConfinstance.Once applied, all settings (and internal variables) are available on the
Celeryappinstance’sapp.confobject.Provided ConfigurationBelow is the configuration that theConfigclass will provide to aCeleryinstance.result_backendDefaults to'db+postgresql://{0.RESULT_DB_USER}:{0.RESULT_DB_PASS}@{0.RESULT_DB_HOST}/{0.RESULT_DB_NAME}'rendered withself.broker_transportDefaults to'sqs’.broker_transport_optionsDefaults to:{'region':'us-west-2','queue_name_prefix':'{}-'.format(QUEUE_NAME_PREFIX)}task_queuesDefaults to the followingsetofkombu.Queueobjects, wherequeuesis the configuration’s internalQUEUESvariable andexchangeis akombu.Exchangeobject constructed from thetask_default_exchangeandtask_default_exchange_typesettings:set([Queue('celery',exchange,routing_key='celery'),Queue(platform_queue,exchange,routing_key='#'),]+[Queue(q_name,exchange,routing_key=q_name)forq_nameinqueues])Note: It is recommended that developers not alter this setting.task_routesDefaults to a function that will generate a dict with therouting_keymatching the value at the first index of a task name
split on the.and theexchangeset to akombu.Exchangeobject constructed from thetask_default_exchangeandtask_default_exchange_typesettingsNote: It is recommended that developers not alter this setting.task_default_exchangeDefaults to'task_exchange'task_default_exchange_typeDefaults to'topic'task_track_startedDefaults toTrue.Internal VariablesBelow are arguments and environmental variables that can be used to
customize the above provided configuration. By convention, all variables
used to construct Celery configuration should should be written entirely
uppercase. Unless otherwise stated, all variables may be specified via
argument or environment variable (with preference given to argument).QUEUESThis should contain an array of names for all service-related queues
used by the Cadasta Platform. These values are used to construct thetask_queuesconfiguration. For the purposes of routing followup
tasks, it’s important that every task consumer is aware of all queues
available. For this reason, if a queue is used by any service worker
then it should be specified within this array. It is not necessary to
include the'celery'or'platform.fifo'queues. Defaults to the
contents of theDEFAULT_QUEUESvariable in the modules`__init__.pyfile </cadasta/workertoolbox/__init__.py>`__.PLATFORM_QUEUE_NAMEDefaults to'platform.fifo'.Note: It is recommended that developers not alter this setting.CHORD_UNLOCK_MAX_RETRIESUsed to set the maximum number of times acelery.chord_unlocktask
may retry before giving up. See celery/celery#2725. Defaults to43200(meaning to give up after 6 hours, assuming the default of the
task’sdefault_retry_delaybeing set to 1 second).SETUP_FILE_LOGGINGControls whether a default logging configuration should be applied to
the application. At a bare minimum, this includes:creating a console log handler forINFOlevel logsa file log handlers forINFOlevel logs, saved toapp.info.loga file log handlers forERRORlevel logs, saved toapp.error.logNote: This may be useful for debugging, however in production it is
recommended to simply log to stdout (as is the default setup of Celery)SETUP_SENTRY_LOGGINGDefaults toTrueif all required environment variables are set,
otherwiseFalse. Controls whetherSentrylogging handlers should be
setup. TheSENTRY_DSNenvironment variable is required for Sentry
logging to be setup automatically. If this condition is met, the
following will be setup:add aSentry signal
handlerto log all failed tasksQUEUE_PREFIXUsed to populate thequeue_name_prefixvalue of the connectionsbroker_transport_options. Defaults to'dev'.RESULT_DB_USERUsed to populate the defaultresult_backendtemplate. Defaults to'cadasta'.RESULT_DB_PASSUsed to populate the defaultresult_backendtemplate. Defaults to'cadasta'.RESULT_DB_HOSTUsed to populate the defaultresult_backendtemplate. Defaults to'localhost'.RESULT_DB_PORTUsed to populate the defaultresult_backendtemplate. Defaults to'cadasta'.RESULT_DB_NAMEUsed to populate the defaultresult_backendtemplate. Defaults'5432'.cadasta.workertoolbox.setup.setup_appAfter the Celery application is provided a configuration object, there
are other steups that must follow to properly configure the application.
For example, the exchanges and queues described in the configuration
must be declared. This function calls those required followup
procedures. Typically, it is called automatically by the`worker_init<http://docs.celeryproject.org/en/latest/userguide/signals.html#worker-init>`__
signal, however it must be called manually by codebases that are run
only as task producers or from within a Python shell.It takes two arguments:app- ACelery()app instance.Requiredthrow- Boolean stipulating if errors should be raise on failed
setup. Otherwise, errors will simply be logged to the module logger
atexceptionlevel.Optional, default: Truecadasta.workertoolbox.tests.build_functional_testsWhen provided with a Celery app instance, this function generates a
suite of functional tests to ensure that the provided application’s
configuration and functionality conforms with the architecture of the
Cadasta asynchronous system.An example, where an instanciated and configuredCelery()app
instance exists in a parallelcelerymodule:fromcadasta.workertoolbox.testsimportbuild_functional_testsfrom.celeryimportappFunctionalTests=build_functional_tests(app)To run these tests, use your standard test runner (e.g.pytest) or
call manually from the command-line:python-munittestpath/to/tests.pyContributingTestingpipinstall-e.pipinstall-rrequirements-test.txt./runtestsDeployingpipinstall-rrequirements-deploy.txtpythonsetup.pytestcleanbuildtagpublish