RevolutionAnalytics/RHadoop — Issue #33: distribute files over hdfs
Author: piccolbo
Created: 2011-12-09 22:18:30+00:00
State: open
Labels: complexity-low, importance-medium, pkg-rmr, type-improvement

to.dfs doesn't partition the files that it creates, which prevents the map phase of a job on the file to use multiple cores. That's all right for I/O bound jobs as to.dfs never creates big data, but for more CPU intensive can be a problem. A simple job with random keys and pass through mapper and reducer can solve this.