mongodb/mongo-ruby-driver — pull request #144 by cheald (2012-12-22 11:45:47+00:00)
Title: Eliminate socket leaks by recycling PoolManager instances

As referenced in [RUBY-522](https://jira.mongodb.org/browse/RUBY-522), ReplicaSetClient#hard_refresh! causes socket and file descriptor leaks by creating new instances of `PoolManager` and storing the old instances (presumably as they may still have active connections), which creates a bunch of `Node` instances, which then stuffs them in the manager's `@members` ivar, never to be released. This ends up burning file descriptors, potentially causing long-running processes to starve the user of available file handles and generally breaking everything once that limit is hit. This also can potentially starve out Mongo clients on other machines by consuming all of the Mongo daemon's connection slots, leaving none for other clients.

This patch alters the behavior of each of the places the would replace a PoolManager by instead simply refreshing it with the new seeds. The general gist is:

1. Pools are now tracked on the PoolManager and re-used where possible
2. Nodes are tracked and re-used when possible
3. A new method, `#refresh!`, accepts a list of additional seeds, unions it with the manager's current seed list, and uses that seed list to refresh the view of the replica set by calling `#connect`.
4. On `#connect`, each active Node has its configuration refreshed, and the pools are rebuilt from the refreshed node data.

`PoolManager#connect` should now be idempotent, ensuring that even if called multiple times, a) existing pools (and any open connections they have) will not be destroyed, and b) any new seeds added to the manager will be used to re-map the replica set. Disconnected or stale nodes are removed, closed pools are purged, and existing healthy nodes and pools are persisted, reducing overhead.

You can test the before-and-after behavior with the following bug replication test:

require 'mongo'

$client = Mongo::MongoReplicaSetClient.new(['192.168.4.139:27021'], :pool_size => 5)

while true
begin
$stderr.puts $client["test"]["test"].find()
rescue
ensure
$stderr.puts "#" * 80
$stderr.puts "%s open file handles" % `lsof -p #{Process.pid} | grep 27021 | wc -l`.strip
end
end

In this test, `192.168.4.139:27021` is a replica set member with another member configured which is not currently up. This causes the replica set connection to fail and invokes a `hard_refresh!`; under the current master, this causes an ever-increasing number of sockets to be opened and maintained. After the patch, this remains constant at 1 socket open.

`test`, `test:replica_set`, and `test:sharded_cluster` suites all pass under ruby-1.9.3, ruby-1.8.7, and jruby-1.7.0.

I think that we should probably have unit tests that prevent regressions of this fix, by causing hard_refreshes and ensuring that the open socket count does not balloon out of control, but the current test suite's structure is something of a puzzle to me, and I'm not sure where to start with it. Advice in that regard would be appreciated.

This is a relatively substantial change, and while I'm reasonably confident that it is performing as expected, careful overview by someone more familiar with the driver's architecture would be appreciated.