dataiap/dataiap — pull request #3 by myw (2012-01-20 08:42:51+00:00)
Title: Cleaned up version of TF-IDF calculation that runs on AWS

It's not quite as clean as I wanted to make it, and there are a few things to note:

* The code should be invoked something like this:

```sh
python mr_tfidf_per_sender_aws.py \
-r emr --num-ec2-instances=40 \
--python-archive=package.tar.gz \
-o 's3://dataiap-wolfsonm-student55-testbucket/tf_idfs_clean' \
--no-output \
--idf_loc='dataiap-wolfsonm-student55-testbucket/idf_parts' \
--aws_access_key_id=$AWS_ACCESS_KEY_ID \
--aws_secret_access_key=$AWS_SECRET_ACCESS_KEY \
's3://dataiap-enron-json/*.json'
```
* You _cannot_ have `s3://` in the `--idf_loc` argument. I'm not sure why: I think it's because then, `mrjob` treats it like some file that it tries to load for you automatically and fails. Debugging this took _hours_, and I couldn't believe what the final solution was.
* There's still no way to get the access key id and secret access key onto the node, since the environment changes. It may be cleaner to do it with the `mrjob.conf` file, but that doesn't work for our purposes either. Anyway, what I've done is kind of a hack, but it works fine and is flexible enough to work. The only thing that's really "broken" about the code is that there are parameters that are really required which are passed in as arguments (ostensibly optional).
* No need to add `boto` to the `package.tar.gz`