mozilla/configman — pull request #11 by twobraids (2011-12-14 22:38:44+00:00)
Title: added config as context for auto factory closing

My original intent was for these configuration objects to hold more than just static constants. As I've mentioned before, I've envisioned factories as part of configuration. For example, rather than just having a collection of constants representing the parameters needed to connect to a database, just offer an object that will hand out database connections on demand.

Aggregations are the basis for constructing factories. The 'trans_generator' (now renamed 'advanced-demo1') was the first of these. While that was an interesting exercise, it was really pretty simple minded. In playing around and implementing a more robust factory I realize that configman was missing a key component to handling factories. We've got the initialization part with Aggregations, but we need tear down, too.

I've implemented this experiment using 'contextlib'. If we think of a configuration as a context that has a scope of a 'with' statement, we've got the mechanism for an orderly tear down of factory objects. When the code block of a 'with' statement ends, we've got the opportunity to implement tear down.

Look at the 'context' function within the ConfigurationManager class. When the context goes out of scope, the '_walk_and_close' function walks the config object trying the 'close' method on every item. Config options that don't have a 'close' method ignore it. However, for those that do understand, it's the opportunity to perform cleanup work. The example 'advanced-demo2.py' demonstrates a database connection pool that is guaranteed to close all its cached connection at the appropriate time.

There are two database classes in the example. The first (default) implements a database connection factory with no pooling. Connections are created, used and destroyed. This is directly applicable to how we use database connections in Socorro when backed with PGBouncer.

Alternatively, we could pool connections locally if we were to choose not to use PGBouncer. I'd expect smaller deployments of Socorro to pool connections this way.

To see the two strategies in action run 'advanced-demo2' in these two ways:

$ ./advanced_demo2.py

**** First query will succeed
FakeDatabaseConnection - created
FakeDatabaseConnection - yep, we did your query
FakeDatabaseConnection - commit
PGTransaction - requestng connection to close
FakeDatabaseConnection - closed connection

**** Second query will fail
FakeDatabaseConnection - created
FakeDatabaseConnection - yep, we did your query
FakeDatabaseConnection - rollback
PGTransaction - requestng connection to close
FakeDatabaseConnection - closed connection

**** about to leave the config context

Notice how the connections get closed after each use. Next try it with the pooling object:

$ ./advanced_demo2.py --database=PGPooledTransaction

PGPooledTransaction - setting up connection pool

**** First query will succeed
FakeDatabaseConnection - created
FakeDatabaseConnection - yep, we did your query
FakeDatabaseConnection - commit
PGPooledTransaction - refusing to close connection

**** Second query will fail
FakeDatabaseConnection - yep, we did your query
FakeDatabaseConnection - rollback
PGPooledTransaction - refusing to close connection

**** about to leave the config context
PGPooledTransaction - shutting down connection pool
FakeDatabaseConnection - closed connection
PGPooledTransaction - connection MainThread closed

In this run, the opportunity to close the connection after each transaction is refused. Only at the end, at the close of the 'with' does the pool close the connections.

Implementation of this feature required some changes to how the 'config' property works. Since we're allowing configuration to contain factories, configuration has state. So rather than caching and doling out a single copy of the configuration, each request will result in a new copy of the DotDict based config object. This allows for more than one copy of configuration to be in use at the same time. I'm thinking ahead to threaded programs where threads may have their own independent copiese.

Try it out and give me feedback.