Package: pyramid_scheduler
Thepyramid-schedulerpackage is a pyramid plugin that allows
asynchronous and deferred task scheduling and management. It usesAPSchedulerfor actual task management andKombufor messaging
between processes.IMPORTANT: the pyramid-scheduler is very new, but fully
functional. The following features are considered “beta”, and
should probably not be used in production:Multiple queues. For now, just use a single queue (and thus you are
limited to a single background process).Intuitive sequence of job events. Currently, the events that can be
listened to (from pyramid_scheduler.api.Event, such as JOB_CREATED,
JOB_EXECUTED, JOB_CANCELED, and JOB_REMOVED) do not reliably fire in
an intuitive order. For example, you may get a JOB_REMOVED event
before the JOB_EXECUTED event for a deferred job.TL;DRInstall:$pipinstallpyramid-schedulerUse:# ini file settings:#   [app:main]#   scheduler.combined   = true | false   ## should execution be in-process?#   scheduler.queues     = jobs           ## space-separated list of queues#   scheduler.broker.url = %(dburl)s      ## the URL used for kombu messaging#   ## other optional settings:#   ##   scheduler.housekeeping#   ##   scheduler.housekeeping.append#   ##   scheduler.jobstore.default.class#   ##   scheduler.misfire_grace_time# enabling the plugin adds a `scheduler` attribute to the registrydefmain(global_config,**settings):# ... (the usual pyramid startup calls) ...config.include('pyramid_scheduler)# create an asynchronous taskdefslow_process(name,id):# ...a slow asynchronous job...defhandle_request_quickly(request):request.registry.scheduler.add_async_job(slow_process,args=('my-first-arg',2))# schedule a deferred task for one hour from nowdefdelayed_process():# ...something that should happen later...defhandle_request_now(request):importtimerequest.registry.scheduler.add_date_job(delayed_process,time.time()+3600)# do something every 10 minutesdefinterval_process(reason=None):# ...gets executed every 10 minutes with an optional reason...defhandle_request_often(request):request.registry.scheduler.add_date_job(interval_process,minutes=10)ConceptsThe pyramid-scheduler package refers to asynchronous or deferred tasks
to be managed as “jobs” and these fall into the following categories:Asynchronous jobs:These jobs are executed immediately, but asynchronously.Deferred jobs, i.e. “date” jobs:Jobs that are scheduled to be performed at a specific time.Interval jobs:Similar to deferred jobs, but that are then re-executed on an
interval.Cron jobs:Similar to interval jobs, but they use a scheduling definition
that is similar to a unix “cron” definition.Conceptually, there are two activities: the activity of creating the
jobs and the activity of executing the jobs. These can be performed by
the same process (with “scheduler.combined=true”), or they can be
performed by different processes (with “scheduler.combined=false”).If there are multiple processes that are creating jobs (for example,
if your are running multiple servers or your WSGI configuration uses
multiple processes), then youCANNOTrun pyramid-scheduler in
combined mode since then there will be multiple processes executing
the jobs leading to multiple executions.Typically, combined mode is used during development where a singlepserveinstance will be used. Then, in production mode, you will
have multiple servers and WSGI processes that generate jobs, that
are then executed by a single background process (managed via thepschedulerdaemon).Pyramid-scheduler supports multiple “queues”. The main reason to use
separate queues is that thepschedulercan be configured to only
process jobs for specific queues, which means that multiple
pschedulers can work in parallel as long as they are listening to
different queues. (A later enhancement is planned to allow multiple
pschedulers to handle a single queue.)IMPORTANT: currently, the following limitations exist:The callback handler provided as the first argument to the scheduler
add_*_job() methods must be a normal module-level defined
function. Itcannotbe a lambda function, an internal function, a
method, or otherwise function that is not globally resolvable using
standard dot-notation.Theargsandkwargsparameters must all be completely
pickle-able.Deferred jobs that are scheduled to occur further in the past thanmisfire_grace_timewill be silently dropped.Jobs that take adateorstart_dateparameter can specify those
values either as an epoch int or float or as a datetime object. If
a datetime is provided, itmustbe timezone “naive” (see the
documentation ofdatetime).Under the hood, pyramid-scheduler uses APScheduler to do the actual
processing and scheduling. For messaging between the job creators and
the pscheduler background process, it uses Kombu messaging, which
supports a variety of transports including Redis and SQLAlchemy. This
package was developed as an alternative to celery, due to severe
limitations found in the celery API and shortcomings in the actual
implementation.InstallationYou can manually install it by running:$pipinstallpyramid-schedulerHowever, a better approach is to use standard python distribution
utilities, and add pyramid-scheduler as a dependency in your project’sinstall_requiresparameter in yoursetup.py. Then run apython setup.py develop.Then, enable the package either in your INI file via:pyramid.includes = pyramid_scheduleror in code in your package’s application initialization via:defmain(global_config,**settings):# ...config.include('pyramid_scheduler')# ...ConfigurationThe following configuration options (placed in the “[app:main]”
section of your INI file):TODO: add documentationscheduler.combinedscheduler.queuesscheduler.delegatorscheduler.broker.urlscheduler.broker.serializerscheduler.broker.compressorscheduler.housekeepingscheduler.housekeeping.appendscheduler.jobstore.default.classscheduler.misfire_grace_timeDebuggingThe first step in debugging a pyramid-scheduler instance is to elevate
the logging, and that is easiest via the application
configuration. Here, an example that increases logging to DEBUG level
and sends the logs to STDERR:[loggers]keys=scheduler, ...[handlers]keys=console, ...[formatters]keys=generic, ...[logger_scheduler]level=DEBUGhandlers=consolequalname=pyramid_scheduler[handler_console]class=StreamHandlerargs=(sys.stderr,)level=NOTSETformatter=generic[formatter_generic]format=%(levelname)-5.5s [%(name)s] %(message)sIf that does not expose the source of the problem, you can take some
of the following actions:Confirm CommunicationYou can confirm that the task producers and consumers are
communicating by sending aprint-jobsmessage. First, check
the configurations by sending the message from a fake producer
by using thepscheduler--messagefeature as follows:$pscheduler--message'print-jobs'{CONFIG}.iniDEBUG[pyramid_scheduler.pscheduler]loadingapplicationfrom"{CONFIG}.ini"DEBUG[pyramid_scheduler.broker]sendingmessage<pyramid_scheduler.api.Eventmessage={'message':'print-jobs'}>tomessengerand you should see something like this in the pscheduler daemon logs
(depending on what happens to STDOUT, you may only see the DEBUG
messages, not the actual Jobstore messages):DEBUG [pyramid_scheduler.broker] received message: <pyramid_scheduler.api.Event message={'message': 'print-jobs'}>
DEBUG [pyramid_scheduler.scheduler] received message event: print-jobs
Jobstore default:
    pyramid_scheduler_wrapper (trigger: cron[hour='0', minute='5'], next run at: 2014-12-03 00:05:00)
Jobstore internal.transient.8524480f-26b4-4a69-8bcd-3bb180d0cf9e:
    housekeeping (trigger: interval[1 day, 0:00:00], next run at: 2014-12-03 14:45:26.696818)If that does not work, you need to check the application
configurations, both on the consumer and producer sides. You may also
need to debug theKombusub-system.TODO: add documentationTask Execution ProcessThere are several ways that the tasks in a queue can actually be
executed. The preferred way, described here, is to use thepschedulerscript which is intended to be run in daemon mode by a
daemon-running service, such as DJB’sdaemontoolspackage.Install daemontools (adjust for your package manager):$apt-getinstalldaemontoolsThis should install & start thesvscanmonitoring against the/etc/servicedirectory. You can do apsto confirm this, and if
it is not running, read the daemontools docs. If it is scanning a
directory other than/etc/service, adjust the following examples
appropriately.Create & configure the logging subsystem by creating the following
file in/etc/service/pscheduler/log/run(wherepschedulercan
be whatever you want). This example will store up to 100MiB of logs
in the/var/log/pschedulerdirectory:#!/bin/sh
exec multilog t s10485760 n10 /var/log/pschedulerCreate & configure thepschedulerservice by creating the following
file in/etc/service/pscheduler/run(wherepschedulercan be
whatever you want). This example will runpscheduleras thewww-datauser (it is simplest if it runs as the same user as the
appserver that is producing pyramid-scheduler tasks):#!/bin/sh
exec env -i PATH="/bin:/usr/bin:$PATH" \
setuidgid www-data \
/path/to/virtualenv/bin/pscheduler \
  /path/to/config.ini \
2>&1And ensure that both files are executable:$chmod755/etc/service/pscheduler/log/run$chmod755/etc/service/pscheduler/run