dssg/census-communities-usa — pull request #8 by evz (2013-07-27 11:44:16+00:00)
Title: Grouping rows from CSV files into smaller chunks before shoving them into Mongo

OK, This time I let this sucker run until my VM was out of disk space just to make sure that it was actually working. Basically I added an iterator that groups the rows from the CSV files into chunks of 10,000 and then shoves them into Mongo.

The issue that I think you're going to run into right away with this is that you're probably going to run out of disk space pretty quickly unless you [disable journalling in Mongo](http://docs.mongodb.org/manual/core/journaling/), at least while you're loading the data. Once the data is loaded, you can probably leave it off, actually, because, as far as I know, it only really concerns durability of write operations. I loaded Alabama as a seemingly average state (and conveniently located at the beginning of the alphabet) and this is what I get when I look at the disk usage for the directory where mongo stores the data on the VM where this was running:

```
$ du -a -h -c /var/lib/mongodb/
1.1G /var/lib/mongodb/journal/j._2
4.0K /var/lib/mongodb/journal/lsn
1.1G /var/lib/mongodb/journal/prealloc.2
1.1G /var/lib/mongodb/journal/j._1
3.1G /var/lib/mongodb/journal
4.0K /var/lib/mongodb/mongod.lock
2.0G /var/lib/mongodb/census.6
4.0K /var/lib/mongodb/_tmp
2.0G /var/lib/mongodb/census.8
65M /var/lib/mongodb/census.0
2.0G /var/lib/mongodb/census.5
129M /var/lib/mongodb/census.1
1.1G /var/lib/mongodb/census.4
257M /var/lib/mongodb/census.2
513M /var/lib/mongodb/census.3
17M /var/lib/mongodb/census.ns
17M /var/lib/mongodb/local.ns
2.0G /var/lib/mongodb/census.7
64M /var/lib/mongodb/local.0
2.0G /var/lib/mongodb/census.9
16G /var/lib/mongodb/
16G total
```

According to [the census bureau](http://www.census.gov/geo/maps-data/data/tallies/census_block_tally.html), Alabama has ~250,000 census blocks and there are ~11,000,000 in the US. Without the journal files (representing about 6.4GB of the total there) you can figure on about 10GB per 250,000 census blocks, right? This is loading all of the jobs data and the geographic crosswalk table (which is very useful and will basically the only way to create larger aggregates of info down the line). Which is rapidly approaching 500GB worth of Mongo data. So, give yourself 1TB (which I think is as big as you can make an EBS volume) and call it good. Once I started grouping the rows from the CSV into chunks of 10,000 I didn't have any memory issues using a VM that only had 1GB worth of RAM available to it.

Oh, one other thing to mention, I added a way to load one state at a time. Just pass in a command line argument after the script like so: `python fetch_and_load.py al`. I'm pretty sure the abbreviation needs to be lower case because that's how all the various paths to the files that are being downloaded are getting built. If you don't give it an argument, it'll just attempt to load the whole enchilada.