OpenTSDB/opentsdb — pull request #379 by jmangs (2014-08-08 22:22:36+00:00)
Title: Added option to timeout sockets server-side. (#304)

By default, Netty sets socket timeouts to 0; this causes sockets to linger forever until they're explicitly closed. As seen in #304, it seems that sockets can be left in a weird state (TIMED_WAIT) when a TSD loses its connection to all of its region servers when `tsd.network.async_io = true` - my guess it has to be something to do with async I/O and re-using sockets `tsd.network.reuseaddress = true` causes some weirdness to occur in Netty.

This PR provides a workaround by providing a way to configure the socket timeout for Netty by using the IdleStateHandler:

http://docs.jboss.org/netty/3.2/api/org/jboss/netty/handler/timeout/IdleStateHandler.html

The `IdleStateHandler` watches the Netty channel and fires an `IdleStateEvent` when N number of seconds have passed without any reads/writes being done on the socket (effectively idle) - I've also updated `ConnectionManager` to inherit from `IdleStateAwareChannelHandler` and implements `channelIdle`.

By default, socket timeouts are left to infinite wait (a timeout of 0 prevents the `IdleState.ALL_IDLE` event from firing - this maintains the current functionality in 2.0) but you can configure them by setting the `tsd.core.socket.timeout` configuration property:

```
# timeout sockets in Netty when idle for 5 minutes
tsd.core.socket.timeout = 300
```

This is what it looks like in the logs:

```
2014-08-08 14:27:53,118 INFO [New I/O worker #1] HttpQuery: [id: 0xf086015a, /0:0:0:0:0:0:0:1:64884 => /0:0:0:0:0:0:0:1:4242] HTTP /stats done in 78ms
2014-08-08 14:32:53,095 DEBUG [New I/O worker #1] ConnectionManager: Closed idle socket.
```

Using this, you can force to sockets to close if they're left open and idle for a long enough duration. This should mitigate the open sockets in #304 when using async I/O while giving the option to maintain the indefinite socket timeouts by default.