Package: ntorque
# nTorque - web hook task queue[nTorque][] is a [task](http://www.celeryproject.org)[queue](https://github.com/resque/resque) service that uses [web hooks][].It is free, open source software [released into the public domain][] thatyou can use from any programming language (that speaks HTTP) to queueup and reliably execute idempotent tasks. For example, in Python:```pythonimport osimport requestsparams = {'url': 'http://example.com/myhooks/send_email'}data = {'user_id': 1234}endpoint = os.environ.get('NTORQUE_URL')response = requests.post(endpoint, data=data, params=params)```[nTorque]: http://ntorque.com[web hooks]: http://timothyfitz.com/2009/02/09/what-webhooks-are-and-why-you-should-care/[released into the public domain]: http://unlicense.org/UNLICENSE## RationalenTorque is designed to be a good solution when you need more reliability thanfire-and-forget but you don't need an [AMPQ][] / [ESB][] sledgehammer to crackyour "do this later" nut.Because it uses web hooks, you can:1. use it from (and to integrate) applications written in any language1. use DNS / web server load balancing to distribute tasks1. bootstrap your task execution environment the way you bootstrap a webapplication -- i.e.: once at startup, re-using your web app's configurationand middleware[AMPQ]: http://www.rabbitmq.com[ESB]: http://en.wikipedia.org/wiki/Enterprise_service_bus## FunctionalitynTorque provides the following endpoints:* `POST /` to enqueue a task* `GET /tasks/:id` to view task statusAnd the following features:* persistent task storage* non-blocking, concurrent task execution* HTTPS and redirect support* configurable (linear or exponential) backoff## ImplementationnTorque is a Python application comprising of a web application and one or moreworker processes. These use a [PostgreSQL][] database to persist tasks and a[Redis][] database as a notification channel.<pre><code>+------+  |  +--------+    +--------+    +--------+  ||POST /|     |Frontend|    |Web app |    |Postgres||------|  |  |--------|    |--------|    |--------|  ||- url |+- ->|- auth  |+-->|- store |+-->|- tasks ||- data|  |  |- rate  |    |- notify|    |        |  ||      |     |  limits|    |        |    |        |+------+  |  +--------+    +--------+    +--------++           ^    +    ||                    |           |   urlrpush        get  data  ||                    |           |    |v           +    v    ||                 +--------+   +--------+     +---------+|Redis   |   |Worker  |  |  |Web hook ||                 +--------+   |--------|     |---------||         |- POST  |+-|->|- perform|| nTorque            +-blpop-> |  data  |     |  task   |+--------+  |  +---------+</code></pre>In the event of a response with status code:* 200 or 201: the task is marked as successfully completed* 202 - 499: the task is marked as failed and is not retried* 500 (or network error): the task is retried[Hack here][] if you'd like a different strategy.[Hack here]: https://github.com/thruflo/ntorque/blob/master/src/ntorque/work/perform.py#L133## AlgorithmThe real crux of nTorque is a trade-off between request timeout and retry delay.It's worth understanding this before deploying -- and how to simply mitigateit by a) specifying an appropriate default timeout and b) overriding this asnecessary on a task by task basis.Like [RQ][] and [Resque][], nTorque uses Redis as a push messaging channel. Arequest comes in, a notification is `rpush`d onto a channel and `blpop`d off.This means that tasks are executed immediately, with a nice evented / pushnotification pattern.Unlike [RQ][] and [Resque][], nTorque doesn't trust Redis as a persistence layer.Instead, it relies on good-old-fashioned PostgreSQL: the first thing nTorque doeswhen a new task arrives is write it to disk. It then notifies a consumer processusing Redis [BLPOP][]. The consumer then reads the data from disk and performsthe task by making an HTTP request to its url.In most cases, this request will succeed, the task will be marked as completedand no more needs to be done. However, this won't happen *every time*, e.g.: whenthere's a network error or the webhook server is temporarily down. Because thereare edge case failure scenarios where the web hook response is unreliable, nTorquerefuses to rely on it as the source of truth&trade; about a task's status. Instead,the single source of truth is the PostgreSQL database.This is achieved by automatically setting a task to retry every time it's read("acquired") from the database. Specifically, the query that reads the task datais performed within a transaction that also updates the task's due date and retrycount. This means that in any failure scenario, nTorque can always just be restarted(potentially on a new server as long as it connects to the same database) and youcan be sure that tasks will be performed at least once no matter where they werein the pipeline when whatever it was fell over.Incidentally, tasks due to be retried are picked up by a background process thatpolls the database every `NTORQUE_REQUEUE_INTERVAL` seconds.More importantly, and where this description has been heading, is the relationbetween the due date of the task as it lies, gloriously in repose, and thetimeout of the web hook call. For there is one thing we don't want to do, andthat is keep retrying tasks before they've had a chance to complete.In order to prevent this behaviour, we impose a simple constraint:> **The due date set when the task is transactionally read and incremented mustbe longer than the web hook timeout.**This means that, in the worst case (when a web hook request does timeout orfail to respond), you must wait for the full timeout duration before your taskis retried. So whilst you may naturally want to set a relatively high timeoutfor long running tasks, you may want to keep it shorter for simper tasks likesending your new user's welcome or reset password email: so that they'reretried faster.The good news is that, in addition to the global `NTORQUE_DEFAULT_TIMEOUT`configuration variable, you can set an appropriate timeout for different tasksusing the [`timeout` query parameter](#usage-api/post).Simple -- once you know how the system works.[BLPOP]: http://redis.io/commands/blpop[Gevent]: http://www.gevent.org[PostgreSQL]: http://www.postgresql.org[Redis]: http://redis.io[Resque]: https://github.com/resque/resque[RQ]: http://python-rq.org/## InstallationClone the repo, install the Python app using:bash pip_install.shYou need Redis and Postgres running. If necessary, create the database:createdb -T template0 -E UTF8 ntorqueIf you like, install Foreman, to run the multiple processes, using:bundle installRun the migrations:foreman run alembic upgrade headBootstrap an app (if you'd like to authenticate access with an API key):foreman run python alembic/scripts/create_application.py --name YOURAPPYou should then be able to:foreman startAlternatively, skip the Foreman stuff and run the commands listed in `Processes`manually / using a Docker / Chef / init.d wrapper. Or push to Heroku, run themigrations and it should just work.## ConfigurationAlgorithm / Behaviour:* `NTORQUE_BACKOFF`: `exponential` (default) or `linear`* `NTORQUE_CLEANUP_AFTER_DAYS`: how many days to leave tasks in the db for, defaultsto `7`* `NTORQUE_DEFAULT_TIMEOUT`: how long, in seconds, to wait before treating a webhook request as having failed -- defaults to `60` see the algorithm sectionabove for details* `NTORQUE_MIN_DUE_DELAY`: minimum delay before retrying -- don't set any lowerthan `2`* `NTORQUE_MAX_DUE_DELAY`: maximum retry delay -- defaults to `7200` but youshould make sure its longer than `NTORQUE_DEFAULT_TIMEOUT`* `NTORQUE_MAX_RETRIES`: how many attempts before giving up on a task -- defaultsto `36`* `NTORQUE_REQUEUE_INTERVAL`: how often, in seconds, to poll the database fortasks to requeue -- defaults to 5* `NTORQUE_TRANSIENT_REQUEST_ERRORS`: 4xx errors which ntorque should retry -- defaults to '408,423,429,449'Deployment:* `NTORQUE_AUTHENTICATE`: whether to require authentication; defaults to `True`-- see authentication section in Usage below* `NTORQUE_ENABLE_HSTS`: set this to `True` if you're using [HSTS][]* `HSTS_PROTOCOL_HEADER`: set this to, e.g.: `X-Forwarded-Proto` if you're runningbehind an https proxy frontend (see [pyramid_hsts][] for more details)* `MODE`: if set to `development` this will run [Gunicorn][] in watch mode (so the appserver restarts when a Python file changes) and will raise HTTP exceptions in theAPI views (rather than returning them). If set to `production` it will run Gunicornbehind a [newrelic][] client. If this isn't quite what you want then either don'tset it or set it to any other string (or hack the `run.sh` and / or `gunicorn.py`scripts)Redis:* `NTORQUE_REDIS_CHANNEL`: name of your Redis list used as a notification channel;defaults to `ntorque`* `REDIS_URL`, etc.: see [pyramid_redis][] for details on how to configure yourRedis connectionDatabase:* `DATABASE_URL`, defaults to `postgresql:///ntorque`* `SQLALCHEMY_MAX_OVERFLOW`, `SQLALCHEMY_POOL_CLASS`, `SQLALCHEMY_POOL_SIZE` and`SQLALCHEMY_POOL_RECYCLE` -- see the SQLAlchemy docs on [engine configuration][]and [pyramid_basemodel][] for more information; if you don't provide thesethen SQLAlchemy will use sensible defaults, also note that if you're using[pgbouncer][] you should set `SQLALCHEMY_POOL_CLASS=sqlalchemy.pool.NullPool`[engine configuration]: http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html[gunicorn]: http://gunicorn.org[hsts]: http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security[newrelic]: https://addons.heroku.com/newrelic[pgbouncer]: https://wiki.postgresql.org/wiki/PgBouncer[pyramid_basemodel]: https://github.com/thruflo/pyramid_basemodel[pyramid_hsts]: https://github.com/thruflo/pyramid_hsts[pyramid_redis]: https://github.com/thruflo/pyramid_redis## Usage / API### AuthenticationIf you set `NTORQUE_AUTHENTICATE` to `True` then you need to create at least oneapplication (e.g.: using the `alembic/scripts/create_application.py` script) andprovide its api key in the `NTORQUE_API_KEY` header when enqueuing a task.### `POST /`To enqueue a task, make a POST request to the root path of your nTorqueinstallation.**Required**:* a `url` query parameter; this is the url to your web hook that you want nTorqueto call to perform your task**Optional**:* a `method` query parameter; which http method to use when calling the webhook --the default is POST, but you can alternatively specify DELETE, PUT or PATCH.* a `timeout` query parameter; how long, in seconds, to wait before treating theweb hook call as having timed out -- see the Algorithm section above for context**Data**:This aside, you can pass through any POST data, encoded as any content type youlike. The data, content type and character encoding will be passed on in the POST(or DELETE, PUT or PATCH) request to your web hook.**Headers**:Aside from the content type, length and charset headers, derived from yourrequest, you can specify headers to pass through to your web hook, by prefixingthe header name with `NTORQUE-PASSTHROUGH-`. So, for example, to pass througha `FOO: Bar` header, you would provide `NTORQUE-PASSTHROUGH-FOO: Bar` in yourrequest headers.**Response**:You should receive a 201 response with the url to the task in the `Location`header.### `GET /task/:id`Returns a JSON data dict with status information about a task.#### `POST /task/:id/push`Pushes a task onto the redis notification channel to be consumed, aquired andperformed. You should *not* normally need to use this. It's exposed as anoptimisation for [hybrid][] integrations.[hybrid]: https://github.com/thruflo/ntorque/blob/master/src/ntorque/client.py#L141## Pro-TipsnTorque is a system for reliably calling web hook task handlers: not forimplementing them. You are responsible for implementing and exposing your ownweb hooks. In most languages and frameworks this is simple, e.g.: in Rubyusing [Sinatra][]:```rubypost '/hooks/foo' do# your code hereend```Or in Python using [Flask][]:```python@app.route('/hooks/foo', methods=['POST'])def foo():# your code here```Key things to bear in mind are:[Sinatra]: http://www.sinatrarb.com[Flask]: http://flask.pocoo.org#### Status CodeAfter successfully performing their task, your web hooks are expected to returnan HTTP response with a `200` or `201` status code. If not, nTorque will keepretrying the task.#### Avoid TimeoutsYour web server must be configured with a high enough timeout to allow tasksenough time to complete. If not, you may be responding with an error when tasksare actually being performed successfully.For example, for a 30 minute timeout with [Apache][] as a proxy:```textTimeout 1800ProxyTimeout 1800```Or with [Nginx][]:```textsend_timeout 1800;proxy_send_timeout 1800;```[Apache]: http://httpd.apache.org[Nginx]: http://nginx.org#### Secure Public HooksIf your web hooks are exposed on a public IP, you are likely to want to securethem, e.g.: using HTTPS and an authentication credential like an API key.It's also worth noting that you may need to turn off [CSRF validation][].[CSRF validation]: http://en.wikipedia.org/wiki/Cross-site_request_forgery#Prevention## SupportRaise [bugs / issues on GitHub](https://github.com/thruflo/ntorque/issues).