resque/resque — pull request #1136 by jonashuckestein (2013-10-08 23:57:57+00:00)
Title: Add RESQUE_PRE_TERM_TIMEOUT

Setting `RESQUE_PRE_TERM_TIMEOUT=n` instructs resque to wait for `n` seconds before sending a SIGTERM to the child process. This allows short-running tasks such as sending an email to complete naturally, before sending a SIGTERM to the child, which causes a `Resque::TermException` and drops the stack so that the worker has no other choice but to re-enqueue. `RESQUE_PRE_TERM_TIMEOUT` defaults to 0, which is equivalent to the current behavior.

Without this change, even short-running tasks were regularly interrupted when workers were scaled down. Simply waiting a little bit before sending SIGTERM to the worker would usually have allowed the task to complete.

This is how things play out for us on heroku, but I'm sure it's useful in other environments, as well:

1. Heroku sends the worker a SIGTERM
2. The worker waits for `RESQUE_PRE_TERM_TIMEOUT` seconds (7) before before sending a SIGTERM to the child
3. The worker sends SIGTERM to the child if it's still running and the child runs some cleanup/re-enqueue code
4. If the cleanup code doesn't finish within `RESQUE_TERM_TIMEOUT` seconds (2), Resque kills it and winds down the worker
5. If for some reason the worker is still alive after 10 seconds, heroku kills it

Is there anything we're missing or any reason not to include this feature in RESQUE? I'm happy to make a pull request for the master branch, as well.