NICTA/scoobi — Issue #9: WordCount is broken due to its confusion of local filesystem with Hadoop FS
Author: benwing
Created: 2011-11-14 21:04:57+00:00
State: open

Your WordCount example is broken because it attempts to create a file on the local filesystem and then pass it to Scoobi, which will naturally use Hadoop's filesystem API (which may or may not refer to the local filesystem). This makes the example break by default on the TACC cluster I'm using, since its default Hadoop FS is HDFS. Now, it happens on my cluster that my home directory is mounted on all the machines, so if I use the option '-fs file:///', everything works. But that's not a general solution for multiple reasons:

(1) There's no guarantee all clusters will be set up with the local filesystem visible on all machines.
(2) There might be good reasons for using HDFS -- e.g. say we were trying to do something like what WordCount does, but with really large files.

Although I haven't looked too closely at the guts of Scoobi yet, I don't see any way to do the "right thing" here -- which is to use the Hadoop API for creating files. The problem is that Scoobi doesn't seem to provide any way of getting a Configuration object so that a Hadoop FileSystem object can be retrieved. Either withHadoopArgs() needs to be changed so that it passes both a list of arguments and a Configuration object (or similar), or another, related function needs to be created, e.g. withHadoopArgsAndConf(). I'd vote for the former for simplicity.

ben