Package: cronq
A cron-like system to run your application tasks across any node, instead of one special snowflake. This is done by keeping your tasks in MySQL and publishing them over AMQP to workers that will run your tasks and eventually save the results back into the DB. This was started as a hackathon project atSeatGeekRequirementsPython 2.7RabbitMQ 3.xMySQL 5.xInstallationUsing PIP via PyPi:pip install cronqUsing PIP via Github:pip install git+git://github.com/seatgeek/cronq.git@0.27.0#egg=cronqAdding to yourrequirements.txtfile (runpip install-rrequirements.txtafterwards):git+ssh://git@github.com/seatgeek/python-cronq.git@0.27.0#egg=cronqUsageThere are various workers that are used by Cronq, as well as a web admin.cronq-runnerTherunnerexecutes tasks, and should be run on hosts that will actually perform work. There is no limit to the number of runners that can execute.The runner requires/var/log/cronq/to exist and be writable by the user executing the runner. If it is not, logs will be written to/tmp.# setup rabbitmq connection infoexportRABBITMQ_URL=amqp://guest:guest@localhost/# specify the rabbitmq queue to listen toexportCRONQ_QUEUE=cronq_jobs# `cronq_jobs` is the default queue# log job output to stdout *as well as* /var/log/cronqexportCRONQ_RUNNER_LOG_TO_STDOUT=1# run commandscronq-runnerWhen run,cronq-runnerwill:Setup a rabbitmq connectionListen to thecronq_jobsqueueRetrieve commands from the queuePublish a message saying the command is startedRun the command in a shell subprocessPublish a message on success/failure to thecronqexchange andcronq_resultsqueue. This is not configurable.Listen for more messagescronq-injectorThecronq-injectorcommand will non-destructively create any necessarycronqtables, though it will need a database to perform this action against. Please note that if you do not have tables created, it is helpful to run the injector first.Theinjectoris used to retrieve jobs from the database and publish them to AMQP. Jobs are published in the following format:# where job is a database record{'name':job.name,'command':unicode(job.command),'id':job.id,}You can ostensibly run as many injectors as necessary. MySQL isolation levels are used to attain locks on job records.# setup rabbitmq connection infoexportRABBITMQ_URL=amqp://guest:guest@localhost/# specify the database connection stringexportDATABASE_URL=mysql+pymysql://cronq:cronq@localhost/cronq# run the comand injectorcronq-injectorcronq-injectorperform a 1 second sleep between job injections, but may perform an unlimited number of job injections in that time period.Note that jobs are not queued up at theexacttime you specify in the database. Rather, jobs that matches the following heuristic are queued one-at-a-time until no jobs are left to be queued for that injection cycle:Job.next_run < NOW() OR Job.run_now = 1cronq-resultsTheresultsaggregator listens to thecronq_resultsqueue for the results ofcronq-runnerexecutions. You can run as many of these as possible, as they will retrieve results one-at-a-time from rabbitmq.# setup rabbitmq connection infoexportRABBITMQ_URL=amqp://guest:guest@localhost/# specify the database connection stringexportDATABASE_URL=mysql+pymysql://cronq:cronq@localhost/cronq# run the results-aggregatorcronq-resultsThese results can be viewed for particular commands within the web-admin, or by inspecting the database.cronq-prunerAs time goes on, you will wish to prune your database of old events. A naive pruner is packaged with cronq, and it has the ability to prune events by a range or to keep at mostNrecent eventscronq-webThe web view is a WSGI app run fromcronq.web:appand requires only database access. The following is an example for running the web admin using webscale technologies:# install libevent-devsudoapt-getinstalllibevent-dev# install required python librariessudopipinstallgreenletgeventgunicorn# specify the database connection stringexportDATABASE_URL=mysql+pymysql://cronq:cronq@localhost/cronq# if you have an aggregated log dashboard, you can provide a search url
# template. it will be used in the web dashboard for linking to logs
# the following strings will be replaced:
#
# {job_id} : replaced with the job's job_id
# {run_id} : replaced with the job's run_id
# {start_time} : replaced with job's start time in ISO format url quoted
#                ex - "2016-08-15T08%3A00%3A11.000Z"
# {end_time} : replaced with job's end time, or now if job is still running, in ISO format url quotedex-"2016-08-15T08%3A00%3A31.999Z"exportCRONQ_LOG_URL_TEMPLATE="https://logs.service/search?run_id={run_id}&from={start_time}&to={end_time}"# run the web admingunicorn--access-logfile--w2--bind:5000--worker-classgeventcronq:make_application\(\)--error-logfile---log-file-# access the panel on http://127.0.0.1:5000The web admin will list available commands, their result history, and a button to allow you to immediately schedule a job.Categories ApiThe web admin exposes acategoryendpoint which allows you to replace a set of jobs with a single API callcurl-v'localhost:5000/api/category/example'-f-XPUT-H'content-type: application/json'-d'
{
    "category": "example",
    "jobs": [{
        "name": "Test Job",
        "schedule": "R/2013-05-29T00:00:00/PT1M",
        "command": "sleep 10",
        "routing_key": "slow"
    }]
}'This adds / updates a job namedTest Jobin theexamplecategory. The time format is ISO 8601. Any jobs no longer defined for the example category will be removed. This allows you to script job additions / removes in your VCS.Configuration ValidationBefore posting cronq configuration, you may wish to validate that the file is in the correct format. To do so, you can use the includedcronq-validatorconsole script.# looks for a cronq.config in the current directory by defaultcronq-validator# can be scoped to a specific filecronq-validator--configcronq.jsonThis will check both json syntax as well as whether the contents pass the Cronqjsonschema. We also include acronq.validatormodule which contains the following methods:cronq_schema: Returns the jsonschema as a python dictionaryvalidate: Takes a path to a config file and runs validation. May raiseValidationError.LicenseBSD