mozilla/socorro — pull request #2077 by twobraids (2014-06-03 21:07:17+00:00)
Title: Fixes bug 1010353 - back to basics - middleware simplification

DON'T PANIC

all that is old is new again, it is time to go back to the future...

this is a "whole cloth" proposal for a dramatic simplification of the middleware. If we're not ready to take this whole thing in one great swallow, then we can break it up into smaller changes. However, I want this to be considered as whole because it solves a lot of problems:

remember the old middleware? This is the old middleware resurrected in a new more flexible form. It fully leverages configman, pushing the complexity to that layer, leaving a cleaner, more easily understandable programmers api. It also totally eliminates hard coding of resources: when a new storage scheme comes around, this version of the middle requires no recoding at all. Ceph will just plug in as easily as it does with the crashmovers. Any service offered by this middleware can be independently configured to use whatever implementations are appropriate. That finally puts the middleware on par with rest of the backend systems like collector, movers and processor.

Just jump to middleware_app.py to see how simple the app becomes:

What does this accomplish:
* all old dependencies on deprecated code are removed. No more Hamiltonian path of dependencies leading to schema.py.
* all interfaces/classes/methods comply with a single unified API - the method naming pattern is now consistent throughout all the methods that implement middleware services.
* package contents are now understandable at a glance - as modules that comply with the service API are _always_ tagged with "_service" in the name. (The PG directory was a mess of mysteries modules with no clear purpose without reading the code. Now we can see what implements a service and what doesn't).
* no service needs to be hard coded to depend on another. Many services use the PG Util service, but hard coded the dependency. As we changed implementations, the PriorityJob service ended up using a baroque injection of instance variables into a dynamically generated class in order for the CrashData service to find it. Now configman offers a lookup to any and all services in the config. Need the Util service? find it at ```self.config.services.Util.cls```
* No more dynamically generated classes - all service wrapping is accomplished with simple single path inheritance
* global resource configuration: all resources can be configured globally using the same resource and secrets sections that the rest of the system uses. That means, unlike the current middleware, the ```+include``` files that we've created for use with the other backend systems can now be used.
* resource exceptions: every service has its own section in the config. If you don't touch them, then the
config will default to whatever was globally set in the ```resource``` section. If you need, for example, the crontabber state service to access PG as a different user / different database - you can override the global config in the ```[[crontabber_state]]``` section.
* All connection based resources (PG, ES, RMQ, HB, Ceph) are accessed through the connection_context/transaction_executor system. Especially for PG, this guarantees correct transactional behaviors. Services just need to declare what they need in RequiredConfig
* All tests of services now use a unified method of generating testing config via ```socorro.unitest.middleware.setup_configman``` -- any test requiring mocking of resources may use config to inject those mocks into tests via configman.
* No dependency on ```.../unittest/config/commonconfig.py```
* locality of reference. The configuration requirements for services should be included with those sevices. The current middleware puts ES config into the middleware_app code. Any other app wishing to use ES must maintain its own independent copy. As requirements evolve, keeping them in sync _will_ be a problem. This PR moves all config into the services that use them - we only need to maintain one copy and it is right there in the code.
* All services have been unified in coding style: PEP 8 & LARS 7 (lots of dividing lines) standards. All files involved in the PR have been PyFlaked and unused code & imports removed.
* confusing use of the the term ```context``` has been removed: sometimes ```context``` meant the config, other times it meant the connection to a resource. All references to config are now ```config```.

What does this lose us?
* no more dynamic service implementation overrides via URL - it seems we rarely ever did this and never in production.