wildfly/wildfly — pull request #6237 by bstansberry (2014-05-06 20:16:32+00:00)
Title: [WFLY-2741][WFLY-3058] Management op administrative cancellation and timeouts

Supersedes https://github.com/wildfly/wildfly/pull/6206 by continuing on with further work related to operation cancellation.

See description of #6206 for discussion of the operation timeout piece (the 1st two commits).

This PR adds commits related to administratively canceling problematic ops.

Caveat: it is still not possible to force MSC stability if some service will not properly return from start/stop. See https://issues.jboss.org/browse/MSC-143. But the cancellation fixes included here will allow the HC processes to no longer block waiting for servers to stabilize, making it possible for an admin to take corrective action (e.g. kill the hung server.)

The last 3 commits do the following:

1) Expose a resource for each currently executing management operation:

[domain@localhost:9990 /] /host=master/core-service=management/service=management-operations:read-children-resources(child-type=active-operation)
{
"outcome" => "success",
"result" => {"-63170310" => {
"access-mechanism" => "undefined",
"address" => [
("host" => "master"),
("core-service" => "management"),
("service" => "management-operations")
],
"caller-thread" => "management-handler-thread - 3",
"cancelled" => false,
"exclusive-running-time" => -1L,
"execution-status" => "executing",
"operation" => "read-children-resources",
"running-time" => 3534000L
}}
}

Strip /host=master off the address for the standalone server address; include server= for the domain server address.

2) Include a "cancel" op in those resources, which if invoked interrupts the thread executing the op. Canceling an op is done via thread interruption, as has been the case since 7.0.

3) Implement a number of fixes related to ensuring that canceling an op via interruption is properly propagated throughout a managed domain, while also ensuring that the thread interruption that is the basic mechanism for cancellation does not result in inadvertently closing the communication channels between the processes in the domain.

4) The ModelControllerClient async API has always supported canceling op from the client side by canceling the Future returned from executeAsync. This is now more reliable in a domain due to 3) above.

5) The parent resource for the active-operation resources includes two convenience operations:

a) find-non-progressing-operation: Checks for an operation that has been holding the exclusive operation execution lock for greater than the provided number of seconds, and if found return its id.

b) cancel-non-progressing-operation: Like find-non-progressing-operation, but instead of returning an operation's id, it cancels that operation.

The intent of these two ops is to simplify identifying/canceling the problem operation that is hanging with the exclusive lock while a number of other ops are blocking waiting for that lock.