airlift/airlift — pull request #220 by frsyuki (2014-08-05 08:20:23+00:00)
Title: Allow external tools to suspend Discovery Announcer through JMX to detach corrupted or shutting down service providers

We're using Presto and airlift.discovery. If we restart a presto-worker (discovery service provider) process, tasks running on the provider fail. To prevent this problem, we need to re-create the entire cluster.
This patch allows us to detach a service provider (presto-worker) so that service users (presto-coordinator) don't send new tasks to the provider any more. Once all existent tasks complete, we can restart the service provider safely.

Automation tool of the graceful shutdown process works as following (pseudo code):

```
jmx = JMX::MBean.create_connection host: "localhost", port: 3000
jmx.find_by_name("io.airlift.discovery.client:name=Announcer").setSuspendAnnouncing(true)
taskManager = jmx.find_by_name("com.facebook.presto.execution:name=TaskManager")
while taskManager.TaskManagementExecutor.QueuedTaskCount > 0 do
sleep 1
end
Process.kill :TERM, pid
```

Although this change itself is incomplete for graceful restart, I think having suspending functionality at Announcer is a first step.