konklone/oversight — pull request #2 by konklone (2014-06-09 02:27:01+00:00)
Title: Load IG reports into Elasticsearch

This is my first attempt at a stand-alone loading script for loading IG reports into Elasticsearch.

It's my first time doing real ETL in Node. Some things:

* I'm using [async](https://github.com/caolan/async) to load every report into Elasticsearch in a predictable order, and to only end the script once they're all done. I don't know how awfully I'd have had to structure this without `async`. Even still, it does force me to create an array of intermediary objects to iterate over, and a standalone callback to operate each one on. I'm not sure whether that's good or bad.
* Making a standalone script, even one that smartly parses command line arguments, is so easy. substack's [minimist](https://github.com/substack/minimist) is a one-line joy. Loading config details directly from `config.js` (and requiring it relative to the file, not the CWD) is easy. Instantiating an Elasticsearch client is easy. I honestly wouldn't mind copy/pasting that boilerplate atop each task, instead of factoring them out into helpers. That's nice.
* The config.js pattern is helpful and simple. I used this in [past projects](https://github.com/isitchristmas/web) and it's the best way to go.
* The official elasticsearch client for JS, [elasticsearch-js](https://github.com/elasticsearch/elasticsearch-js), seems nice, straightforward, and has a nice logging feature.

I'm not operating with any models yet or anything. Overall, Node is kind of like the opposite of Rails: Rails does a great job of framework-izing the basic boilerplate of managing MVC and task management. Node makes it all moot by encouraging small and modular everything.

Where Ruby emphasizes building DSLs that take large problems and map them closely to your brain so you can manage them effectively, Node somehow makes the problems so small that you don't need any extra help.

Todo:

- [ ] Read the accompanying `.txt` file and load that into Elasticsearch as well.