tidepool-org/tideline — pull request #170 by jebeck (2014-09-05 20:17:53+00:00)
Title: runtime data validation

As discussed with @cheddar, the done criteria for this first task is to create a schema that validates against the current output of the "preprocess" plugin (which is to be rewritten as nurse-shark). This schema is done and ready for review. To test, you can inspect the console logs in the tideline example, which now includes information about the number of items validated (should be all!):

![screenshot 2014-09-05 13 14 47](https://cloud.githubusercontent.com/assets/1588547/4171034/57e2d910-3539-11e4-9586-58c8da351500.png)

Testing against demo data and a `blip-output.json` (exported from blip via the `window.downloadJSON()` function available in the console) is recommended :) (Do `export DATA=blip-output.json` to use the real exported data in the example.)

Since "preprocess" is about to be rewritten as nurse-shark (for the next task), I'm only calling this validation in tideline's internal example app, as I expect to have a discussion with @cheddar and @nicolashery (at least) about where and how to integrate validation in blip after the move to nurse-shark, especially given the fact that this validation is somewhat expensive, time-wise. On demo data (which spans 30 days and is our eventual target for data paging), the validation takes around 800ms on the command line, 2000ms in the browser. On real data (spanning 180 days currently), it takes ~4s on the command line, ~12s in the browser(!) There are multiple ways we could get around this expensiveness. One is validation via async stream processing. Another would be not including validation in the production environment. I think a better idea would be including validation via query parameter (in all environments), so that we can provide a link somewhere to ask a user who is concerned about strange or missing data to click and be able to pass along to us information about the results of the validation. But all that discussion is for later - just wanted to point things out now while they are fresh in my mind.