rails/rails — pull request #17549 by dylanahsmith (2014-11-07 00:17:10+00:00)
Title: activerecord: Avoid filling up connection pool on reconnect failure.

cc @Sirupsen who discovered this bug
cc @byroot

## Problem

If a connection to the database is lost during a request, then the ConnectionManagement middleware will call `ActiveRecord::Base.clear_active_connections!` to release the connection to the connection pool, which will result in a reconnect when the next request tries to obtain the connection from the pool. If the reconnect fails, then the raised exception prevents the exception from being associated with the current thread, but remains in the connection pool.

This happens because the call to `checkout` in [ConnectionPool#connection](https://github.com/dylanahsmith/rails/blob/99bb2fd892a2a876ba0f4017bd3cc87033a4deb3/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#L269-L275) prevents the connection from being added to `@reserved_connections`.

```ruby
def connection
# this is correctly done double-checked locking
# (ThreadSafe::Cache's lookups have volatile semantics)
@reserved_connections[current_connection_id] || synchronize do
@reserved_connections[current_connection_id] ||= checkout
end
end
```

That connection will still be in the connection pool in `@connections`, so will count towards the connection count, but will not be in the `@available` connections queue, so the connection will never be checked back out of the pool. The connection will also not be `reap`ed, since `conn.owner.alive?` will still be true.

If the connection to the database is temporarily lost enough times, then the connection pool will fill up.

The same problem can occur with using ConnectionPool#checkout directly, since the exception will prevent the connection from being returned to the caller, so ConnectionPool#checkin won't be called for that connection.

## Solution

Catch the exception in ConnectionPool#checkout_and_verify so that the connection gets returned from ConnectionPool#checkout, and will get added to `@reserved_connections` when using ConnectionPool#connection.

The returned connection won't be active, but each subsequent call to `ActiveRecord::Base.clear_active_connections!` will give it another opportunity to reconnect when it is checked out of the pool again, so will gracefully recover when the connection can once again be established to the database.