NICTA/scoobi — pull request #286 by blever (2013-07-17 01:54:00+00:00)
Title: Make BridgeStoreIterator lazy.

Fix BridgeStoreIterator to construct SequenceFileReader objects lazily
using a Stream. Fixes #285.

BridgeStoreIterator constructs multiple SequenceFileReader objects under
the hood for each part file underlying a materialised DList. These
objects are constructed eagerly and consume file descriptors (socket
connections to HDFS). If the number of part files exhausts the available
file descriptors for the client-side Scoobi process, bad things ensure.

Add a Spec that without this fix shoud fail in cluster mode (but will
still pass in in-memory and local mode).