cloudera/crunch — pull request #9 by dhgarrette (2012-03-13 09:55:23+00:00)
Title: Prevent PTable.materialize from pulling entire contents into memory

The method `crunch.PCollection.materialize` is careful to return an `Iterable` that 1) doesn't run the hadoop job until the data is needed and 2) keeps the result data where it is (eg, in a file) until it is needed. Unfortunately, the way `PTable.materialize` was written, a call to `materialize` would always run the job immediately and load the entire result into memory (as a `List`).

The issue was that `map` was being called on the `Iterable` returned from `crunch.PCollection.materialize`, and `map` on an `Iterable` always iterates through immediately, applying the transformation and returning a new `Iterable`. Thus, the original, smart, `Iterable` is lost and replaced with a simple `List`.

The solution to this problem is to use an `Iterator` since `map` in this case will be delayed until the data is accessed.

This change gives us what we want: calling `PTable.materialize` returns an `Iterator[(K,V)]` such that the hadoop job is not run until the data is accessed, and the data isn't ever fully loaded into memory. This `Iterable` can be accessed repeatedly and the hadoop job will not re-run: the results will simply be re-accessed from their original location.

Also, the explicit call to `iterableAsScalaIterable` is unnecessary since the conversion will be handled by the implicit in `scala.collection.JavaConversions`.