apache/spark — pull request #2866 by JoshRosen (2014-10-21 01:43:14+00:00)
Title: [SPARK-4019] Fix MapStatus compression bug that could lead to empty results

This commit fixes a bug in MapStatus that could cause jobs to wrongly return
empty results if those jobs contained stages with more than 2000 partitions
where most of those partitions were empty.

For jobs with > 2000 partitions, MapStatus uses HighlyCompressedMapStatus,
which only stores the average size of blocks. If the average block size is
zero, then this will cause all blocks to be reported as empty, causing
BlockFetcherIterator to mistakenly skip them.

For example, this would return an empty result:

sc.makeRDD(0 until 10, 1000).repartition(2001).collect()

The root problem here is that MapStatus has a (previously undocumented)
correctness property that was violated by HighlyCompressedMapStatus:

If a block is non-empty, then getSizeForBlock must be non-zero.

I fixed this by introducing a new SparseCompressedMapStatus which only stores
the sizes of non-empty blocks. I also added new tests and assertions.