apache/spark — pull request #3486 by ksakellis (2014-11-26 19:40:19+00:00)
Title: [WIP] SPARK-2450 Adds exeuctor log links to Web UI

Creates a general way to add log links to the executor
page in the web UI.

This is achieved by:

1) Adding two new spark listener events: onExecutorAdded and onExecutorRemoved(this is only for symmetry). The events carry some executor information including log url locations.

2) Utilizing environment variables to pass the log urls to the executor backend. The prefix is: SPARK_LOG_URL_

Since each cluster manager exposes logs in different ways, we use the environment variable prefix as described above to prevent us from subclassing the CoarseGrainedExecutorBacked.

This is a work in progress and as such, there are minimal unit tests. In addition log urls are only propagated for yarn-client and yarn-cluster mode. With minimal work, adding log url propagation for standalone mode is also possible.