pulp/pulp — pull request #1019 by bmbouter (2014-06-23 22:59:17+00:00)
Title: Added delay-and-log support if mongo is not available

Adds a behavior to all Pulp services and httpd to not crash if mongo is not available. If mongo is not available, log it as an error, wait 3 seconds, and try again. This code also included an audit of the existing reconnect support for mongo. You could already restart mongo, and Pulp will recover when it comes back online. I have tested this, and audited the use of pymongo throughout the Pulp codebase.

To test this feature, stop all pulp services including mongo. Ensure that Qpid (or rabbitMQ) is running. Start any pulp_* service or httpd and observe the log. Then, start mongod and observe the services pick up and are functional. All previous logging behavior also should continue to work, which was challenging with this code.

This test does not come with tests for the server/pulp/server/db/connection.py file. That file has no unit tests, which is concerning since it sets up database connections. I have filed that as a [BZ ](https://bugzilla.redhat.com/show_bug.cgi?id=1112419). All other code should have unit test coverage.