reddmetrics/forma-clj — pull request #388 by robinkraft (2013-07-18 00:25:39+00:00)
Title: updated fires processing workflow

Addresses #387. Fires preprocessing now produces a complete fire series up through the present instead of truncating it during preprocessing using `est-start` and `est-end`. The problem with dropped pixels that we saw in #387 was caused by an obscure bug in the interaction between `PreprocessFire` and date filtering in `forma-tap`. The solution is to preserve the fires as full time series and trim the fire series to the length of the trends series within `forma-tap`. This PR is a beast, but it fundamentally changes how fires processing happens, and frankly most of the work was in the tests.

This fix works, because ... well ... to make a long story not very short, the fire series truncated to the estimation dates in preprocessing were being merged with series of trends values that started in `2005-12-19` and ran through the most recent period for which they'd been calculated. `forma-seq`, used in `forma-tap`, doesn't know about dates, and tries to merge fires and trends into a `FormaValue` based on the length of the fire and trends series. If there's no fire for a given date, you get a `FormaValue` with trends values and 0 fires for that date. Given a short fire series (e.g. update only for the last month) and a long trends series, the fires were only being joined with the first few trends values. The rest of the trends values would get dropped.

Since we're now filtering `FormaValues` by date in updates (it's a major speed up to neighbor processing), and since the fires were incorrectly joined with the early trends values, there were no pixel-date tuples that fell within the date range specified by `est-start` and `est-end`. So any pixels with fires were getting dropped.

Phew! Does that make any sense at all? If not, well, this fixes the issue, so life is good.