Yelp/firefly — pull request #61 by fhats (2013-10-09 23:32:15+00:00)
Title: Multiple worker processes and data source exclusions from UI

This patch provides two improvements, the rationales for which are intertwined. Firefly's `AggregatingDataSource` needs to be able to aggregate data sources in the same data server as itself. Doing this currently would cause the `AggregatingDataSource` to block on itself when trying to issue HTTP requests against the blocked data server. This solves that problem by adding multiple worker processes to the data server. I added multiple processes to the data server because it solves this problem and it is much, much less susceptible to the concurrency issues that SQLite is prone to in multi-process environments due to the fact that annotation posting is the only activity that causes writes to the data server's sqlite database. This is *not* the case for the UI server, where SQLite updates happen much more frequently and need more robust concurrency protections.

I also added a configuration option for `DataSource` to allow it to be omitted from the data server's enumeration of the data sources it knows about. This means that `DataSource`s can be hidden from the user, but can still be accessible by `AggregatingDataSources`. It's kind of an ugly shim, but we need it asap.