HubSpot/java-memcached-client — pull request #3 by eabbott (2014-06-19 03:15:25+00:00)
Title: Added minimum time threshold when triggering connection lost

Currently the spymemcached drivers will detect a lost memcached server in two ways. Both methods appear too slow in practice.

After 996 consecutive timed out memcached calls will trigger a server as lost. The timeout is 1s, so a delay is built in depending on the concurrency. This method of detection will take more or less time depending on the particular traffic profile of the application. Unfortunately even our most highly trafficked service capiweb takes several minutes [to detect failure.](https://hubspot.tv.appneta.com/app/ContactsApiWeb/app_server/visualize#fs=1402992720&fe=1402996320&v=heatmap&xpd=1353390&ts=1402994400000&te=1402994640000&nd=912508&xd=1151943)

The second method of detection is a bit fuzzy and may only exist as an artifact of testing. During functional testing on my laptop/aws with iptables used to simulate a failure, the lost server is detected in 20-40s. This method of detection has not been seen in and environment with real traffic such as qa or production. Even in qa when iptables has been used to simulate failure, the failed server is not detected for several minutes. This implies that load is a factor. Again I will stress the unknown and that we really haven't done many tests yet in qa/prod, so the data is very sparse.

This PR introduces detection determinism based upon the clock. With this PR we can configure a memcached pool to detect a host as failed after X consecutive timed out requests (eg 50) over N millis (eg 30sec). The goal is to detect a memcached failure in a deterministic time period (eg 30s).

As for the code changes themselves, the particular angle taken is one of eventual consistency. The sync aspects are not at all exact, there is leeway for things to be missed immediately, but eventually they will be picked up correctly. To say it more clearly in prose, it's entirely possible that exactness will be missed and detection will happen ms to sec later than it could have. That is deemed an acceptable improvement over the current state of the drivers. In particular I'm referring to TCPMemcachedNodeImpl.continuousTimeout and TCPMemcachedNodeImpl.continuousTimeoutStart being synchronized separately.

Our version of the drivers are 3yrs out of date, the up to date version does not have similar functionality. Its feasible that if this code proves useful that we could submit a PR.

@HiJon89 @axiak @marlier @mjball @wsorenson @jaredstehler