edwardcapriolo/filecrush — pull request #2 by alexanderdean (2013-04-07 10:26:20+00:00)
Title: Add support for filecrushing on Elastic MapReduce

**Working PR - do not pull yet**

Hi @edwardcapriolo - this is an open pull request to add support for using filecrush on EMR.

There are three main things to fix:
1. Instantiating the right type of `FileSystem`
2. Allowing an explicit location to be specified for the `tmpDir` (e.g. `--tmp-dir=s3n://my-emr-processing-bucket/tmp`)
3. Replacing the `fs.makeQualified(dir).toUri().getPath()` pattern with something that doesn't strip important S3 bucket information

#1 is done, see PR. #2 is straightforward - just a bit of extra configuration code. #3 is a bit harder - I am working through this for EMR, but might need some help from you to make sure my changes don't break filecrush on standard HDFS.

Hoping this is the start of a collaboration! We're really excited about filecrush here at Snowplow.