Replace magic lengths with constants in PySpark.

Write the length of the accumulators section up-front rather
than terminating it with a negative length.  I find this
easier to read.
Remove Pickle-wrapping of Java objects in PySpark.

If we support custom serializers, the Python
worker will know what type of input to expect,
so we won't need to wrap Tuple2 and Strings into
pickled tuples and strings.
Add custom serializer support to PySpark.

For now, this only adds MarshalSerializer, but it lays the groundwork
for other supporting custom serializers.  Many of these mechanisms
can also be used to support deserialization of different data formats
sent by Java, such as data encoded by MsgPack.

This also fixes a bug in SparkContext.union().
Send PySpark commands as bytes insetad of strings.
FramedSerializer: _dumps => dumps, _loads => loads.
Misc style changes in the 'yarn' package.
Merge branch 'master' into yarn-cleanup

Conflicts:
	yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
	yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
	yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala
	yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
Added logging of scheduler delays to UI
A few more style fixes in `yarn` package.
Add a "new-yarn" directory in SPARK_HOME, intended to contain Hadoop-2.2 API changes.
Hadoop 2.2 YARN API migration for `SPARK_HOME/new-yarn`
add http timeout for httpbroadcast
[SPARK-963] Wait for SparkListenerBus eventQueue to be empty before checking jobLogger state
Emit warning when task size > 100KB
Removed unused basestring case from dump_stream.
Add optional Hadoop 2.2 settings in sbt build.

If the Hadoop used is version 2.2 or derived from it, then Spark
will be compiled against protobuf-2.5 and a protobuf-2.5 version of
Akka 2.0.5.
Merge remote-tracking branch 'origin/master' into yarn-2.2

Conflicts:
	yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
Merge pull request #212 from markhamstra/SPARK-963

[SPARK-963] Fixed races in JobLoggerSuite
Merge pull request #207 from henrydavidge/master

Log a warning if a task's serialized size is very big

As per Reynold's instructions, we now create a warning level log entry if a task's serialized size is too big. "Too big" is currently defined as 100kb. This warning message is generated at most once for each stage.
Merge pull request #146 from JoshRosen/pyspark-custom-serializers

Custom Serializers for PySpark

This pull request adds support for custom serializers to PySpark.  For now, all Python-transformed (or parallelize()d RDDs) are serialized with the same serializer that's specified when creating SparkContext.

For now, PySpark includes `PickleSerDe` and `MarshalSerDe` classes for using Python's `pickle` and `marshal` serializers.  It's pretty easy to add support for other serializers, although I still need to add instructions on this.

A few notable changes:

- The Scala `PythonRDD` class no longer manipulates Pickled objects; data from `textFile` is written to Python as MUTF-8 strings.  The Python code performs the appropriate bookkeeping to track which deserializer should be used when reading an underlying JavaRDD.  This mechanism could also be used to support other data exchange formats, such as MsgPack.
- Several magic numbers were refactored into constants.
- Batching is implemented by wrapping / decorating an unbatched SerDe.