Relax assumptions on compressors and serializers when batching

This commit introduces an intermediate layer of an input stream on the batch level.
This guards against interference from higher level streams (i.e. compression and
deserialization streams), especially pre-fetching, without specifically targeting
particular libraries (Kryo) and forcing shuffle spill compression to use LZF.
Relax assumptions on compressors and serializers when batching

This commit introduces an intermediate layer of an input stream on the batch level.
This guards against interference from higher level streams (i.e. compression and
deserialization streams), especially pre-fetching, without specifically targeting
particular libraries (Kryo) and forcing shuffle spill compression to use LZF.
Merge branch 'master' of github.com:andrewor14/incubator-spark
Avoid reading the entire batch into memory; also simplify streaming logic

Additionally, address formatting comments.
Typo: phyiscal -> physical
Update docs
Privatize methods
Also privatize fields
Merge github.com:apache/incubator-spark

Conflicts:
	core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
Use JSON for ExecutorsUI
Merge github.com:apache/incubator-spark into persist-ui
Add JSON Serializability for all SparkListenerEvent's

This also involves a clean-up in the way these events are structured. The existing way
in which these events are defined maintains a lot of extraneous information. To avoid
serializing the whole tree of RDD dependencies, for instance, this commit cherry-picks
only the relevant fields. However, this means sacrificing JobLogger's functionality of
tracing the entire RDD tree.

Additionally, this commit also involves minor formatting and naming clean-ups within
the scope of the above changes.
ExecutorUI: render completely from JSON

Additionally, this commit fixes the bug in the local mode, where executor IDs of tasks
do not match those of storage statuses (more detail in ExecutorsUI.scala).

This commit currently does not serialize the SparkListenerEvents yet, but instead
serializes changes to each executor JSON. This is a big TODO in the upcoming commit.
ExecutorUI: Serialize events rather than arbitary executor information

This involves adding a new SparkListenerStorageFetchEvent, and adding JSON serializability
to all of the objects it depends on.
Serialize events both to and from JSON (rather than just to)

This requires every field of every event to be completely reconstructible from its
JSON representation. This commit may contain incomplete state.
Fix bugs for EnvironmentUI and ExecutorsUI

In particular, EnvironmentUI was not rendering until a job begins, and ExecutorsUI
reports an incorrect number (format) of total tasks.
Also deserialize new events

This includes SparkListenerLoadEnvironment and SparkListenerStorageStatusFetch
BlockManagerUI: Add JSON functionality
JobProgressUI: Add JSON functionality

In addition, refactor FileLogger to log in one directory per logger
Add functionality of reconstructing a persisted UI from SparkContext

With this commit, any reconstruct SparkUI resides on default port of 14040 onwards.
Logged events are posted separately from live events, such that the live SparkListeners
are not affected.

This commit also fixes a few JSON de/serialization bugs.