apache/incubator-spark — pull request #533 by andrewor14 (2014-02-04 03:34:58+00:00)
Title: External spilling - generalize batching logic

The existing implementation consists of a hack for Kryo specifically and only works for LZF compression. Introducing an intermediate batch-level stream takes care of pre-fetching and other arbitrary behavior of higher level streams in a more general way.