metamx/druid — pull request #561 by deepujain (2014-05-22 09:42:32+00:00)
Title: Improve ingestion MR job completion time.

For 12GB input data containing 8000 files, there are 8000 mappers because of TextInputFormat and its a overhead. The two MR jobs take around a day to complete.

CombineTextInputFormat instead of TextInputFormat combines multiple splits for a single mapper and reduces the strain on hadoop platform. It greatly improves job completion time as there are fewer number of mappers to bookkeep.

mapreduce.input.fileinputformat.split.maxsize controls amount of data given to a single mapper. Ex: setting it to 67,108,864 (64M) will create a single mapper for 64M of data (hence multiple input files).