18F/analytics-proxy-nodejs — pull request #7 by konklone (2015-01-01 02:57:41+00:00)
Title: WIP: Framework for downloading, processing, and saving GA reports

This adds `reports.json`, `download.js`, and an ignored `data/` directory for output JSON.

This is primarily meant for easy exporting, but is a useful separation even for a proxy - the proxy can create bulk data as it goes, and load that data into Mongo et al separately.

`reports.json` is meant to replace `analytics_urls.txt`. I also moved the `ids` parameter into `config.js`.

`download.js` uses a lot of code from `data.js`. The `Analytics` module in `download.js` should be separated into its own module. There's clear overlap, and we can refactor later and avoid duplication. I'm starting it this way to make a cleaner extract-transform-load workflow, which I think will benefit the proxy.

This PR is huge because I also did an update of dependencies, and removed `node_modules/` from the repo (because it involves a compiled C extension, which will differ between OS X and Ubuntu).

More work to go here -- but you can try it out by:

* checking out this branch
* filling in the `ids` parameter in `config.js` (see `config.js.example` for an example)
* `rm -r node_modules`
* `npm install`
* `node download.js`

You should see something like:

```
$ node download.js

[weekly-users] Fetching...
[weekly-users] Saving original...
[weekly-users] Saving processed...
[weekly-users] Done.
```

And then look in `data/original` and `data/ready` to see the raw GA data, and a processed form, respectively.

TODO:

- [ ] Better schema for the processed data, maybe similar to GDS'.
- [ ] More reports.
- [ ] Documentation
- [ ] Probably some refactoring of other parts to remove duplication, but that could also come post-merge.

Don't bother opening the Changed Files tab for this PR - just visit the [`downloading`](https://github.com/18F/analytics-proxy-nodejs/tree/downloading) branch directly.