stephenh/spark — pull request #1 by stephenh (2013-03-12 17:34:03+00:00)
Title: Close the reader in HadoopRDD as soon as iteration end.

This fixes a bug where HadoopRDD's reader was only closed in the TaskContext completion context.

This is normally fine, but when coalescing a large number of S3 files, the task may not complete until too many readers have been left open, and the S3 connection pool will fill up (resulting in very misleading "connection timeout" exceptions).

This changes the HadoopRDD to call close() both at the end of iteration (when on errors) and also in the TaskContext completion (if there are failures, but will call close again if there are no errors--calling it twice should be fine).

There were several "gotNext"-based iterator implementations in the codebase, each slightly different, so along with fixing this bug, I pulled the common code out into a "NextIterator" helper class, so that each location only needs to implement "getNext".