vanzin/spark — pull request #1 by vanzin (2014-06-03 21:28:52+00:00)
Title:  [SPARK-1768] Make event logger use a single file.

Currently the event logger uses a directory and several files to
describe an app's event log, all but one of which are empty. This
is not very HDFS-friendly, since creating lots of nodes in HDFS
(especially when they don't contain any data) is frowned upon due
to the node metadata being kept in the NameNode's memory.

Instead, all the metadata needed for the app log file can be
encoded in the file name itself. (HDFS is adding extended attributes
which could be used for this, but we need to support older versions.)

This change implements that approach, and also gets rid of FileLogger,
which was only used by EventLoggingListener and the little functionality
it provided can be much more concisely implemented inside the listener
itself.

With the new approach, aside from reducing the load on the NN, there's
also a lot less remote calls needed when reading the log directory.