WorldBank-Transport/open-transit-indicators — pull request #208 by lossyrob (2014-09-25 07:56:54+00:00)
Title: Feature/indicator refactor

- Changed all geotrellis-y names to scala, or opentransit.
- gtfs-parser is now included as a subproject. The tests and opentransit are their own subprojects. There's a testkit to aid the testing on the DB, and a test GTFS network that we'll write be able to write nice clean unit tests against in simpler times.
- Split out all the Spray route stuff.
- Split out the ingesting of GTFS data.
- Implemented a design for the indicators that I hope will promote good vibes and beautiful codes.

The indicator design worked for all the indicators I that were implemented when I started this beast of a refactor. There's some complications that came up when I was merging in the latest changes, namely, indicator calculations needing more information than I was feeding them. I'm getting around this for now by doing a precomputation of what each indicator wants and shoving it through in a params object. I can imagine a better way to do it, we can talk it over.

Indicators are implemented in a map-reduce style, where the map is either over individual trips or the set of trips that make up a route. The intermediate results that come from the map are then reduced down based on the aggregation type the indicator declares it can contribute to.

There's a pipeline of turning an indicator result (route or routeType mapped to doubles, or one Double for the entire system) into the Containers that are used to ship things over to the Django service. There's some weirdness here that comes from the translation of the different types of aggregates into one type of JSON object that django is reading. If the scala code were to save off to the database, some of that weirdness would go away.

I *did not* implement some things, namely the saving of the StopsBuffers, the WeeklyServiceHours indicator, and some of the test. The test *are not* passing. I'm getting different results, and I have a hunch that it's because of the difference in what I included in the TransitSystem which is filtered on dates (namely, I included only those stops of a trip which are inside the time period). We can discuss the right way to do that later. I'm also curious to where the test results were coming from; the dates in the test SamplePeriods were for January 2010, while the calendar.txt and the calendar_dates.txt specifies service times for mid-2014. I was getting no results until I realized I was requesting TransitSystems for times the files didn't specify. So we'll have to work that out.

Another unfortunate thing is that while attempting to split out the indicator tests, I ran into a lot of issues with trying to set the database up once and use it for all tests. I don't know why this was happening, but I decided to stop banging my head on the wall and just import the files per test. This clearly isn't the way to do it, but that can be fixed in another PR.