RevolutionAnalytics/RHadoop — Issue #47: dfs.size
Author: piccolbo
Created: 2012-02-02 23:03:34+00:00
State: open

it would be nice to have a way to test the size of a file (or directory) to be able to set parameters for a job (imagine a sampling rate that's inversely proportional to the size, for instance) or to switch to the local backend when the data sizes don't warrant using hadoop, which happens sometimes in iterative jobs. It would probably be even better to have a record count, but I don't know of any way to get that without a mapreduce job.