Threshold failure detector issue

1) Threshold failure detector, marks a node as available when it
receives a non-catastrophic error after configured number of
catastrophic errors. When a node goes unavailable, generally first there
will be lots of connectExceptions ( catastrophic) and timeouts for
already established connections. This causes failure detector to treat a
node that is being down as up and affects the client latency as all the
get waits for the connection timeout to happen and goes for the serial
requests.
2) Threshold failure detector, marks a node as unavailable after a
window rolls over. Threshold failure detector tracks successes/failures
in window ( default 5 minutes) and if the successes drop below a
configured ( defaul 95 ) percentage and if there are more errors than
configured it marks the node as down. When the window rolls over, and if
the first request succeeds or when a node comes back up the failure
count is not reset. This causes an available node to be marked as down.
3) Enhance the unit tests to cover the above 2 cases.
4) Dump the statistics when a node goes down to reason about it in the
logs.

The new code is written with the following reasoning.

Do the book keeping first.
Make decision next.

The new code intentionally does not reset the catastrophic errors on a
window roll over as it will be reset by the first success anyway. The
node can still flap when the failure is above minimum and the percentage
oscillates around the configured percentage. But I don't see any good
workaround and nor it was an actual issue on 20+ repetitions I created
for an internal repro. So leaving them as it is.

Previous code mixed both of these and lead to many issues. The previous
code left many loose ends ( Boolean represented in int, Generic set
methods, when it required reset, Set methods at individual variable
level, when only reset method is required, copy pasted code).

PS: There is a more serious third issue, Selector drains the parallel
queue. When the request requires connection establishment Selector
handles the connection too. If the node is dead Selector is going to
wait configured time for connection to timeout ( default 5 seconds) . In
this time Selector is not pumping read/write and hence all these
requests eventually time out. We are discussing the potential issues for
this fix and will address this in the next fixes. There are many other
minor issues I uncovered as well.
ClientRequest Executor fix and log fixes

1) Protocol negotiation request does not have a timeout associated with
it . So if the connection fails the Input/Output buffers are leaked and
cause Out Of Memory errors. Setting the timeout for the Protocol
negotiation requests as the caller expects a timeout.

No new logs are introduced. Only the existing log is made more
informative.

2) When protocol negotiation timesout, an obscure error message is thrown,
IllegalStateException. This happens as the code tries to read the result
in case of timeout. Handle this case and report a meaningful error.

3) Serial request times out on a connection, error message does not
mention the socket that timed out.

4) Fix other minor logging information.
Exceptions goes uncounted

Get does parallel requests first and if it fails/times move on to the
serial requests. Exceptions are tracked if the parallel requests fail or
timesout after the pipeline is finished. But if the Serial requests are
processing then the exception goes unnoticed. This causes the node to
stay available for a long time affecting the latency and error rates.

This fix reduces the time window an exception goes unnoticed. I thought
about moving the error handling inside the callback and not have it at 2
places. But not sure what will be the perf impact of the error tracking
code, could be negligible but saving that for a later time.