Package: redbike
# RedbikeA simple, flexible scheduler for pools of distributed workers.Redbike is a helpful building block with intentionally limited features.It's easy to schedule and distribute work however you like.Redbike is available from the Python Package Index:```bash$ pip install redbike```## High LevelRedbike has a simple design, centered around a few data structures in Redis.* A registry of job schedules (a [hash](http://redis.io/topics/data-types#hashes))* A registry of job statuses (another [hash](http://redis.io/topics/data-types#hashes)) format: "EVENT:TIMESTAMP"* A timeline of scheduled work (a [sorted set](http://redis.io/topics/data-types#sorted-sets) scored by timestamp of next run)* Any number of job queues ([lists](http://redis.io/topics/data-types#lists))* A single dispatcher process to enqueue work that is due on the timeline* Expiring keys to keep track of jobs that have started running and not yet returned or timed out.* Any number of worker processes## Configuration```bash$ redbike --config=/etc/redbike.conf ...```Redbike needs to know the worker class, the prefix to use for all ofits keys and the location of the time file (see below). Alsoit needs to know how to connect to Redis. By default it looks in`.redbike.conf` but you can specify any file.Connection information is found in a `[redbike-redis]` stanza andmirrors the arguments of py-redis' `StrictRedis` class. The restof the settings are found in a `[redbike]` stanza.Other information in the config file is ignored, so you can usea shared config as long as it has something like this in it:```ini[redbike]prefix: myappworker: somepackage.somemodule:MyWorkerClass('initstring')timefile: /var/log/myapp.redbike.timefile```In Python:```pythonbike = Redbike(MyWorkerClass('initstring'),prefix='myapp',timefile='/var/log/myapp.redbike.timefile')```## Entering Jobs```bash$ redbike set <JOBID> <SCHEDULE> [--after=<TIMESTAMP>]```To run jobs in redbike just enter a JOBID and schedule. Schedulescan be in iCal [RRULE](http://www.kanzaki.com/docs/ical/rrule.html)format or one of the following special values:* `NOW` - run once right away* `AT:TIMESTAMP` - run once at the specified time* `CONTINUE` - just re-queue right away* `STOP` - don't run anymoreIf `--after` is provided, it will override the timefile (see below)and when (or if) to schedule each job will be based on this value.The set operation does not wait for the dispatcher, and entersthe job directly into the timeline or work queue whereappropriate.In Python:```pythonbike.set('JOB1:A', 'NOW')```## Dispatch and the Time File```bash$ redbike dispatch [<WORKER>] [--schedules=<SCHEDULESCSV> [--after=<TIMESTAMP>]]```Dispatch keeps checking the timeline and places jobs into work queueswhen they are due.The dispatcher continually updates the contents of the timefile withthe current timestamp as it runs. When starting dispatch or settingschedules, the time in the timefile is used to determine when and ifthe next run of the job should occur. When no `--after`argument is provided and no timefile is found, the default value isthe current time.A CSV of JOBID,SHEDULE pairs can also be provided. These scheduleswill be set before dispatch begins.## Consuming Work```bash$ redbike work [<WORKER>]```Redbike consumes jobs from the work queues using the worker class's`queue_names()` method and passes them to the worker's `work()` method.If `StopWork` is raised, the job's schedule is set to `STOP`. If `UnsetJob`is raised, the job will be unset (removed from the rotation).  Ifany other exception is raised it is logged and the job is notrescheduled. If the job takes longer than the timeout it is also notrescheduled. Otherwise the job's schedule is checked and itreturns to the queue or timeline if appropriate.## Writing a WorkerRedbike includes a simple worker class that is flexible enough formany cases. Just override its `work()` method.```python# mymodule.pyfrom redbike import RoundRobin, StopWorkclass Worker(RoundRobin):def work(self, jobid):should_stop = do_something(jobid)if should_stop:raise StopWork("Don't run this job any more!")```This class chooses work queues based on the portion of theJOBID following the last `:`. It is initialized with a colonseparated string specifying the queues to work and the orderin which to work them. If, for instance, you want to put jobsinto two queues and have queue "A" worked twice as often asqueue "B", this would do the trick:```bash$ redbike set job1:A CONTINUE$ redbike set job2:B CONTINUE$ redbike work mymodule:Work("A:A:B")```## BackoffWorkers with `CONTINUE` schedules may sometimes wish tointroduce a backoff. The worker's `work()` method mayreturn an interger value of backoff seconds and the jobwill be entered into the timeline to be enqueued forwork again after the specified backoff has elapsed.## Outstanding JobsWhen workers consume jobs from the queue, they set a keywith an expiration and unique value to mark the occasion. Whena worker completes normally, it removes that key and reschedules.If that key no longer exists or the value is changed, it isconsidered timed-out it will not reschedule. This behaviorprevents the worker from stepping on schedule changes (typicallymade by `set()` and `unset()`) that may have happened while thejob was being worked.## TimeoutsTimeouts can be controlled by your worker class. Just definea `timeout(self, queue_name)` and return an integerfor timeout in seconds or `None` to use the default timeout.The default default timeout is 10 seconds. You can also overridethis in your `[redbike]` config stanza with a `default-timeout`setting.You should set a timeout that your jobs wont overrun unless itis ok for them to overlap.## Stopping```bash$ redbike control HALT```Both dispatcher and worker processes watch for a control keyto be set to `HALT` in Redis and stop immediately after completingtheir current task.## Failure ModesRedbike does its best to fail gracefully but managing the failuremodes of your workload is out of scope. How to approach itdepends on how you feel about things like the same scheduledinstance of a job getting worked twice or getting missed whenthings crash and burn. That said, there are a few commands thatcan help.Jobs which may have fallen out of circulation and need to be resetcan be found by time of their last status.```bash$ redbike statuses [--before=<TIMESTAMP>]JOBID,statusJOBID,status...```Output is in CSV format.Statuses are set throughout the lifecycle of jobs in Redbike.* TML - entered into the timeline* ENQ - entered into a work queue* BAD - failed to schedule due to a bad RRULE* WRK - picked up by a worker* STP - stopped when the worked raise StopWork* DIE - worker raised an unexpected exceptionTo dump the schedules of all the jobs:```bash$ redbike schedulesJOBID,scheduleJOBID,schedule...```Here again, output is in CSV format.Finally, to spy on an individual job:```bash$ redbike tell <JOBID>```This will output a json representation like this:```json{"status": "ENQ:1381720578","next_run": null,"schedule": "CONTINUE"}```## Removing Unwanted JobsOnce a job is no longer relevant and you want to take it out ofrotation:```bash$ redbike unset <JOBID>```The job will complete if it is already running, but will otherwisebe removed and not worked.