rathboma/scoobi — pull request #1 by rathboma (2012-05-09 15:45:38+00:00)
Title: Supporting multiple output filesystems

Hey guys,

This change supports multiple output filesystems seamlessly without the need to override any properties.

However, I don't think my code does this in the right way at all, I'd really like some direction on how to do what I did properly.

Essentially all the change does is modify the getWorkingPath call to get a working directory for the correct output uri scheme, but I had to make a bunch of assumptions about where to get the output path from, and I don't think I did that correctly.

Some questions:

- I don't like how I get the output path (and check the filesystem), is there a way to get a single output path from somewhere?
- I had to map through reducers/sinks to get (multiple?) output paths instead.

- given that there can be multiple outputs, why does scoobi store all results in a single tmp-out directory; shouldn't there be a temp output path per reducer given that there is an output path per reducer?

- I had to hack up the Conf.getWorkingDirectory to accept a uri, this also seems hacky, as there are still other calls to this method that don't get a properly scoped uri, so maybe they will break the feature in certain edge-cases?