basho/yokozuna — pull request #375 by rzezeski (2014-05-09 20:58:48+00:00)
Title: Reduce TIME-WAIT load

This addresses #367

A combination of ibrowse's inefficient load balancing algorithm and
default socket inactivity timeout of 10 seconds can cause TIME-WAIT
load. It becomes worse as the pool size of pipeline sizes are
increased. This patch is a temporary workaround to reduce socket
churn and thus TIME-WAIT load. It does so by increasing the
inactivity timeout to 60 seconds across the board. Below is a chart
of showing the amount of socket churn in connections per minute for
the different timeout values both at idle and while under load from
basho bench. These numbers were calculated by a DTrace script which
counted the number of new connections being accepted on port 8093.

| Timeout | Socket Churn At Idle | Socket Churn Under Load |
|---------|----------------------|-------------------------|
| 10s | ~59 conns/min | ~29 conns/min |
| 60s | ~9 conns/min | ~6 conns/min |

The timeout is set via application env because ibrowse has the
absolute most complex configuration management code I have ever seen
and this was the easiest way to make sure the timeout is set
correctly. This is just a workaround until after 2.0 when other HTTP
clients and pools may be tested. ibrowse seems to have many issues,
this is but just one.

For more background see the following issues:

https://github.com/basho/yokozuna/issues/367
https://github.com/basho/yokozuna/pull/358
https://github.com/basho/yokozuna/issues/330
https://github.com/basho/yokozuna/issues/320