triggering shutdown by setting a redis flag

**Problem Statement**
We use rq on heroku and their way to the shutdown the worker that runs rq does not ensure a safe shutdown.
   
From Heroku docs:  https://devcenter.heroku.com/articles/dynos
```
When a worker is stoped by ps:scaling or pushing out a new release or any heroku operation.
The dyno manager sends the working SIGTERM

The process then has 10 seconds to shut down gracefully.

If the process is still alive -- then SIGKILL is then sent
```

So if a job takes more then 10 secs (which many of our jobs do) -- we are going to be out of luck.  The job will be killed in a potentially unsafe point

So we needed an approach where we could set a flag on redis, that the rq worker can ping.

I introduced the key:  'rq:worker:pause_work'  Which any process can set.  If the worker sees that it has been set -- it hops into a pause loop, until the key is deleted.

In Worker#work at the top of the `while True:` block

``` python
    try:
        before_state = None
        notified = False

        while Worker.paused() and not self.stopped:

            if burst:
                self.log.warn('Paused in burst mode -- exiting.')
                self.log.warn('Note:  There could still be unperformed jobs on the queue')
                raise StopRequested

            if not notified:
                self.log.warn('Stopping on pause request REALLY.')
                before_state = self.get_state()
                self.set_state('paused')
                notified = True
            time.sleep(1)
    except StopRequested:
        break
```


No objections to any of these. Will make these changes in thr morning.

I was also thinking of modifying `info` to give the status of paused. Any
objections?
On Nov 9, 2014 7:56 PM, "Selwin Ong" <notifications@github.com> wrote:

> In rq/worker.py:
>
> > @@ -351,6 +353,29 @@ def work(self, burst=False):
> >          self.set_state('starting')
> >          try:
> >              while True:
> > +                try:
>
> Lastly, can we please move the code for dealing with suspension in a
> separate method? We've been trying to simplify the main while loop for a
> while so I'd like to add as little logic as possible to this part of RQ.
>
> Perhaps we could name it worker.check_for_suspension() - other naming
> suggestions are also welcome :). It should look something like this:
>
> while True:
>     self.check_for_suspension()
> # other code
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/nvie/rq/pull/434/files#r20063512>.
>