apache/spark — pull request #791 by cloud-fan (2014-05-15 11:58:26+00:00)
Title: improve performance of MemoryStore#tryToPut by elimating unnecessary lock

It's unefficient to drop memory blocks to disk inside a synchronized block as IO is slow. As the TODO says, we just need synchronize selecting blocks to be dropped. So my implementation is: in `ensureFreeSpace`, we iterate entries and select blocks to be dropped. But instead of dropping block inside `ensureFreeSpace`, we can just mark selected entries as dropping, and return these blocks, let the caller do the dropping. When other thread call `ensureFreeSpace` again, they will skip entries that marked as dropping when iterating entries. And the caller, `tryToPut`, will do the dropping before put the new block into entries. In this way, we can do dropping in parallel.
As I don't have a server to test large data set, I just sleep 1 second in BlockManager#dropFromMemory and set spark.storage.memoryFraction to 0.06 so that many blocks will be dropped into disk. Result shows my implementation is much better and I hope someone can test this on real case.