eventlet/eventlet — pull request #113 by btmorex (2014-07-26 00:16:05+00:00)
Title: fix pool bug

This fixes a bug where in certain circumstances greenlets can get blocked indefinitely in pool.get(). To trigger this bug, the following conditions must be met:

1.) There are greenlets waiting on items from the pool (pool.waiting() > 0)
2.) pool.create() must throw an exception
3.) pool.get() must not be called again (or more accurately, greenlets will block until pool.get() is called again)

It seems like those would be fairly rare circumstance, especially 3.), but our use of pool triggers this any time we have a database failover. We maintain two different pool's corresponding to database servers db1 an db2. Normally, we would just use the pool of connections to db1. Suppose that db1 goes completely down (no longer responds to network requests):

1.) We'll almost certainly have pool.waiting() > 0 because queries that were taking milliseconds, now must wait on a timeout in the seconds range
2.) pool.create() will throw an exception (some kind of connect timeout)
3.) As soon as we get a network exception, we failover and start using db2's pool and don't call get() on db1's pool again.

So basically, any failover scenario cause a greenlets to block indefinitely.

This fix has two caveats:

1.) You can no longer have a pool of None objects since they now have special meaning. This was already the case for db_pool
2.) If there are greenlet's waiting and create() throws an exception we lose exception info because put() causes a greenlet switch. We raise the exception again, but it now appears to originate in pool itself.