kite-sdk/kite — pull request #30 by gabrielreid (2014-03-06 15:47:42+00:00)
Title: Add type parameter to pathIterator method

When reading a dataset (or a view) through Crunch, Kite currently builds up an iterator of all individual file paths to be read within the view. In the case where there are there are a large number of files (consider partitioning based on date, with 20 files per day over a year, resulting in 7300 files), the resulting job configuration for the Crunch job becomes huge.

Building up the job configuration in Crunch for so many individual input files can be extremely slow (mostly due to the resulting performance of Configuration#substituteVars), and in some cases even causes an IOException because the serialized Configuration's size goes over the limit of 5MB.

This issue is largely mitigated by [CRUNCH-335](https://issues.apache.org/jira/browse/CRUNCH-335), which isn't currently available in a released version of Crunch. Even so, it seems useful to reduce the size of the job configuration when dealing with a large number of input files.

FileInputFormat can work with directories instead of individual file paths, so this patch makes the pathIterator method parameterized so that it's possible to create an iterator of individual files, or an iterator of leaf-level directories. The directory iterator is used by CrunchDatasets, avoiding the issue of creating an oversized job configuration.