jblomo/oddjob — Issue #3: Subclassing TextInputFormat outputs line numbers
Author: jblomo
Created: 2012-01-11 01:49:35+00:00
State: open

Hadoop streaming removes line numbers from TextInputFormat (see PipeMapper.java ignoreKey). Using TextInputFormat subclasses can be confusing to people using streaming with tab separated files because now there is an "extra" key: the line number. To get around this, all oddjob classes could subclass KeyValueTextInputFormat instead. This could be a breaking change, but I think it would make using oddjob with mrjob a more consistent experience.

We're testing the change now and will bump the version number if everything works out.