Yelp/mrjob — Issue #296: local mode's file split emulation is incorrect
Author: davidmarin
Created: 2011-11-29 22:46:04+00:00
State: open
Labels: Bug

In an attempt to simulate Hadoop, local mode can split files so that they can be read by several mappers/reducers.

There are at least two known issues with the split logic:
* It attempts to split `*.gz` and `*.bz2` files, which Hadoop would simply leave as-is.
* Its byte-counting is wrong in Windows because Windows line endings are 2 bytes (we should read/write in binary mode).

These are both fairly benign issues; the first results in files being decompressed earlier than they need to be, and the second results in slightly less splits or less even splits than there should be on Windows.