osmlab/to-fix — pull request #51 by sbma44 (2014-11-14 16:20:42+00:00)
Title: tests / async refactor

This wound up being a bigger lift than I had intended, because I wanted to avoid setTimeout() hacks. Most of the import & prioritization system has been rewritten to be async, which speeds things up a little bit (particularly for small data loads). A useful exercise for me, though probably overkill for a set of import scripts.

A few notes:

- `import-csv.js`'s command line interface has been simplified to only accept arguments rather than also accepting filenames piped to it over stdin. This changes how it should be called -- see `load.sh` for an example.
- I leaned heavily on `async-queue` only to discover that it can easily exhaust the call stack when dealing with `to-fix`ish quantities of tasks -- I had to use `async` for `prioritize.processGeoJSON`. At some point I'd like to go back and kill the `async-queue` dependency entirely, but this revelation arose late in the process.
- During development, weird failing test conditions often had to do with half-formed contents in ldb/ or fixed/. Blowing away the contents of those directories is a recommended first step if you see trouble.
- I have tested the command line import process, but ironically the addition of `tape` tests means that the CLI wrapper code is a bit less thoroughly-tested than it had been. Please do keep an eye out.