The ServerBuilder API in finagle provides options for specifying setting...

...s

(a) channelMaxLifeTime (b) channelMaxIdleDuration and (c) OpenConnectionsThresholds.

Code that implemented these settings was removed as part of the refactorings

* https://github.com/twitter/finagle/commit/04f4d25ddbef0ba01569fe1e117295fe54234c2c
* https://github.com/twitter/finagle/commit/d92fedf49df

Bug fixes that go along with re-introducing the change
* Do not rely the service request completing to add a connection to the queue.
This fails the trivial case of just opening connections initially on startup

* Dont forget to remove the connection from the queue

* Fix another issue with ICF.

This fix  largely addresses the timeout exceptions seen on rediscover last week when using thresholds. Tests results were obtained for upto 200 threads and the difference in performance is marginal when using synchronized queue. As an example

Code without thresolds running 200 threads making 1MM requests

Time taken for tests:   299.657087 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Requests per second:    3337.15 [#/sec] (mean)

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   33 373.3      3   21027
Processing:     0   25  51.7     16    5039
Waiting:        0   13  47.5      6    5034
Total:          0   59 378.1     20   21156

Percentage of the requests served within a certain time (ms)
  50%     20
  66%     28
  75%     34
  80%     38
  90%     51
  95%     82
  98%    207
  99%    543
 100%  21156 (longest request)

Vs

Code with thresholds running 200 threads making 1MM requests

Time taken for tests:   355.871021 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Requests per second:    2810.01 [#/sec] (mean)

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   41 405.7      4   21518
Processing:     0   28  59.5     17    3836
Waiting:        0   15  54.7      7    3819
Total:          0   70 413.6     22   21655

Percentage of the requests served within a certain time (ms)
  50%     22
  66%     31
  75%     38
  80%     44
  90%     67
  95%    134
  98%    283
  99%    847
 100%  21655 (longest request)

* Add a counter to track the number of closed connections