mozilla/socorro — pull request #952 by twobraids (2012-11-16 03:35:10+00:00)
Title: Another NO MERGE trial ini scheme

We've been struggling with how to set up ini files for what I see is a conflicting set of goals:

1) Simply works out of the box for dev
2) uses ini files because they are clear and easier to understand than Python code
3) Separates division of maintenance labor between dev and ops
4) Keeps local config out of the repo

After messing around with this seemingly forever, I'm going to propose a radical simplification:

one ini per app and optionally one ini per external resource. This scheme is simple to understand and simple to implement. Here's how it addresses the issues above.

1) the defaults will be for a stand alone system without using HBase. When deployed for the first time, -dist files are copied to their .ini equivalents. If it is desired to have a production system using HBase, the person doing the installation will have to run each app in turn having it write out an HBase aware ini file. We'll have instructions on how to do this. I believe it it is inevitable that we should create samples of ini files for more complicated configurations.

2) it turns out that ini files aren't as easy as initially thought. We added the +include feature to get more flexibility. Only with that could we even conceive of the separation of maintenance responsibility. We're finding that +include isn't enough and now were talking about optional includes and changing ini to allow for repeating/overriding key/value pairs. If we're going as far as inventing a domain a specific language, we ought to just use Python instead.

3) we've been trying to separate roles in ini file maintenance by using include files. This results in a profusion of ini files because ini files are line order dependent and do not allow repeating keys or sections. This adds only adds complexity (possibly up to eight ini files for the processor alone). In this proposal, I've marked values interesting to ops with two different comments: "OPS - set once" and "OPS - volatile". Let the OPS folks see the whole file rather than a proliferation of disjoint parts. Settings relevant to them are marked as such.

4) it has been argued that separating roles into include flies would mean that potential volatile key/value pairs would not get over written when a new version is deployed. I would suggest that a better solution the installation system should never overwrite an existing ini file when copying -dist files to ini files. If they layout or values in an -dist file change between versions of socorro, then there must be manual tuning of existing ini files anyway (configman doesn't write ini files with includes – they are manual additions). Splitting -dist files into ones that are updatable while others are not is again adding complexity to resolve an infrequent problem.

The install system should copy -dist files to the .../config directory if and only if ini files do not already exist there. The person doing an upgrade would then use the release notes and a diff tool to copy changes into their existing ini files in the .../config directory or /etc/socorro directory.