apache/spark — pull request #1255 by andrewor14 (2014-06-28 06:15:38+00:00)
Title: [SPARK-2307][Reprise] Correctly report RDD blocks on SparkUI

The existing code in `ExecutorPage.scala` requires a linear scan through all the blocks to filter out the uncached ones. Every refresh could be expensive if there are many blocks and many executors.

The proper semantics should be the following: `StorageStatusListener` should contain only block statuses that are cached. This means as soon as a block is unpersisted by any means, its status should be removed. This is reflected in the changes made in `StorageStatusListener.scala`.

The `StorageTab` must also be updated, because it currently detects dropped blocks only if their storage levels change to `StorageLevel.NONE`, which no longer happens, because now we simply remove their statuses.

If you have been following this chain of PRs like @pwendell, you will quickly notice that this reverts the changes in #1249, which reverts the changes in #1080. In other words, we are adding back the changes from #1080, and fixing SPARK-2307 on top of those changes. Please ask questions if you are confused.