NICTA/scoobi — pull request #81 by rathboma (2012-05-09 16:31:04+00:00)
Title: Supporting multiple output filesystems

Hey guys,

This change supports multiple output filesystems seamlessly without the need to override any properties.

However, I don't think my code does this in the right way at all, I'd really like some direction on how to do what I did properly.

Essentially all the change does is modify the getWorkingPath call to get a working directory for the correct output uri scheme, but I had to make a bunch of assumptions about where to get the output path from, and I don't think I did that correctly.

Some questions:

I don't like how I get the output path (and check the filesystem), is there a way to get a single output path from somewhere?

I had to map through reducers/sinks to get (multiple?) output paths instead.
given that there can be multiple outputs, why does scoobi store all results in a single tmp-out directory; shouldn't there be a temp output path per reducer given that there is an output path per reducer?
I had to hack up the Conf.getWorkingDirectory to accept a uri, this also seems hacky, as there are still other calls to this method that don't get a properly scoped uri, so maybe they will break the feature in certain edge-cases?

I'd prefer to set some sort of global uri scheme, but I didn't know the best place to do that.

(this includes the contents of the other pull request, but please feel free to ignore those minor changes)