apache/spark — pull request #3218 by woggle (2014-11-12 02:24:22+00:00)
Title: [WIP][SPARK-4351] RDD cached read logging + display on web UI

This set of patches adds cached RDD block attempted read/write trace logging to TaskMetrics and therefore the event log, and uses this information to display/compute RDD miss rates on the Web UI.

This patch removes the elegance of the contents of the storage web UI being entirely from StorageStatuses (as RDD access records shouldn't go away when a block manager does for the purpose of tracking hit/miss rates); I'd appreciate comments on cleaner solutions.

This still needs some additions to the UI tests, and I'd like to make sure it doesn't do anything bad for perf.