reddmetrics/forma-clj — pull request #358 by dpetrovics (2013-04-18 06:02:06+00:00)
Title: Feature/trends performance

I updated the telescoping-trends function to compute the largest short term drop statistic on just the longest subset of the given timeseries. Before, it was computing the short term drops for each telescoping timeseries, which is pretty inefficient. For example, using the forma_test.clj sample numbers as a metric, we can look at NDVI from 2005-12-31 to 2012-04-22. With the 16 day temporal resolution and start period of 693, this would yield a start-idx of 134 and an end-idx of 280 for the telescoping series. So we would have [[0-134] [0-135] ... [0-280]]. The old method was to compute the short stat for EACH of those subsets. That was in effect taking rolling windows of 30 points at a time for each series, smoothing them, and taking the minimum for each. Thats 147 timeseries, with 106 to 251 windows each, and 97 to 242 rolling averages each. Roughly 50,000 calculations.

The new way just computes these moving average drops for the longest timeseries (0-280 in the above example). It then applies reductions min to the result, and we take the last N values where N is the number of timeseries. Sounds kind of confusing now that I'm typing it out, but everything seems to work nicely. The key is that using just the last telescoping series we can get all the info that we had before, except now its only 251windows+242averages=493 calculations. I assume the calculations are relatively simple, so it remains to be seen how much time this will save in practice, but hopefully it'll shave some down!

@robinkraft , I wasn't sure if you wanted to keep the performance test in there, so I just kept it in. Also, new-trends might be better called something like calculate-trends-no-short or something like that?