tumblr/colossus — pull request #36 by DanSimon (2014-12-02 19:27:38+00:00)
Title: new idleCheck method for ConnectionHandler

Fixes #33 . Currently, Service servers handle connection timeouts by scheduling a message to themselves. Aside from being inefficient, it leads to errors when a worker receives the scheduled message after the connection has already closed.

This adds a new `idleCheck` method to `ConnectionHandler` that is periodically called on every attached connection handler by the worker. The frequency of the check is thus global for the entire worker, and is currently hardcoded, although we can make this configurable in the future.

Also, the frequency has been decreased from 250 milliseconds to 100 milliseconds. I did some benchmarking on how long a single check takes, and for a worker with 500 attached connections it takes only about 100us per check on a MacBook pro, so we're looking at about 0.1% of worker time going towards idle checks. This leads me to believe we probably don't want to make the frequency any higher by default.

The downside here is that means request timeouts for services are now less precise, for example a configured request timeout of 20ms will only be detected after an average of 60ms. However, in most cases specifying timeouts in intervals of 100ms should be sufficient, and eventually we'll make the frequency configurable so that users can set it to whatever they want if they require more precise timeout checking.

Lastly, this doesn't fix service client timeouts, but that's a more involved issue for a separate PR