apache/spark — pull request #895 by tdas (2014-05-27 22:10:18+00:00)
Title: [SPARK-1940] Enabling rolling of executor logs, and automatic cleanup of old executor logs

Currently, in the default log4j configuration, all the executor logs get sent to the file [executor-working-dir]/stderr. This does not all log files to be rolled, so old logs cannot be removed.

Using log4j RollingFileAppender allows log4j logs to be rolled, but all the logs get sent to a different set of files, other than the files stdout and stderr . So the logs are not visible in the Spark web UI any more as Spark web UI only reads the files stdout and stderr. Furthermore, it still does not allow the stdout and stderr to be cleared periodically in case a large amount of stuff gets written to them (e.g. by explicit println inside map function).

This PR solves this by implementing a simple RollingFileAppender within Spark (disabled by default). When enabled (using configuration parameter `spark.executor.rollingLogs.enabled`), the logs can get rolled over by time interval (daily, by default). Old logs (older than 7 days, by default) will get deleted automatically. The web UI can show the logs across the rolled over files.