mesos/spark — pull request #518 by woggling (2013-03-11 21:43:07+00:00)
Title: Send block sizes as longs in BlockManager updates

Currently, UpdateBlockMessage stores the sizes of blocks (in memory and on disk) as longs but serializes them as ints, silently truncating the values. This means that blocks which are larger (or estimated larger) than 2G are accounted for incorrectly, which can result in the master thinking a block manager has more memory free than its capacity.

This patch serializes UpdateBlockMessage sizes as longs.

I assume that the sizes were serialized as ints for some reason, in which case this may create a regression. (I suspect that there are other problems with blocks >2G, e.g. it may not be possible to persist them at MEMORY_ONLY_SER, but these should at least not be silent problems.)