elasticsearch/elasticsearch — pull request #7878 by chaudum (2014-09-25 19:02:05+00:00)
Title: Added graceful shutdown behaviour

We at Crate recently implemented a procedure to gracefully stop nodes moving shards away before finally shutting down in order keep all the data available.
This can be used to perform a rolling upgrade of the cluster.

We've also seen, that there were already some discussions about how to implement that in ES here: https://github.com/elasticsearch/elasticsearch/issues/4248

This PR contains the following changes:

- added new settings called:
`cluster.graceful_stop.min_availability`,
`cluster.graceful_stop.reallocate`,
`cluster.graceful_stop.timeout`,
`cluster.graceful_stop.force`
to control the shutdown behaviour
(documented here: https://crate.io/docs/en/latest/configuration.html#graceful-stop)

- added signal handling to perform a graceful shutdown on receiving the `USR2` signal (the current implementation depends on classes that are only available in the oracle and openjdk VMs)

- added DISABLED state for `LifecycleComponent` and transition methods

The shutdown process works as follows:

1. after sending graceful shutdown signal, all services of InternalNode will be disabled, the current implementation only rejects new HTTP requests on this node as transport connections are usually steady connections

2. depending on the graceful_stop settings listed above the node will deallocate its shards using the AllShardsDeallocator/PrimariesDeallocator classes

3. node shuts down once de-/reallocation is finished or returns back to its prior state on any error (if `cluster.graceful_stop.force` is not set to true)