A first crack at getting a Drakefile in place to be the ETL glue... Still very broken.
Got the nightly downloads working and loading into ElasticSearch via the Drakefile
Redid all processing with the wonderful event-stream lib. Got rudimentary MOD processing started. Now outputting proper ElasticSearch bulk update files with both index and update operations.
Make the record primarily use NTYPE for notice type if it is a MOD
Adding a new script that converts stdin XML (in our grants format) to JSON stdout
Moving format-bulk script into a common dir. Can be used on new grants JSON output for loading into Elasticsearch.
Fixing a couple of bugs in format-bulk and removing the old one under FBO
Adding assert.sh as a submodule
Adding first integration test to check FBO bulk loads into Elasticsearch.
Extract setup stuff into a reusable file
Adding test for FBO bulk formatting.
Fix filename typo
Adding test for bulk formatting grants data
Adding (partial) new test for MOD loading and description appending
is_mod accidentally got into the bulk sample files. internal field-- removing
Adding a TODO note
Cleanup
Removing Solr client code from grants loader
A tweak to a sample file and a test file
Adding package.json to common