Implementing SPARK-878 for PySpark: adding zip and egg files to context and passing it down to workers which add these to their sys.path
Removed the unused shuffleId in ShuffleDependency's constructor.
Removed the mapSideCombine option in CoGroupedRDD.
Removed the mapSideCombine option in partitionBy.
Moved shuffle serializer setting from a constructor parameter to a setSerializer method in various RDDs that involve shuffle operations.
Merge pull request #846 from rxin/rdd

Two minor RDD refactoring
Merge pull request #840 from AndreSchumacher/zipegg

Implementing SPARK-878 for PySpark: adding zip and egg files to context ...
Initial work towards scheduler refactoring:

- Replace use of hostPort vs host in Task.preferredLocations with a
  TaskLocation class that contains either an executorId and a host or
  just a host. This is part of a bigger effort to eliminate hostPort
  based data structures and just use executorID, since the hostPort vs
  host stuff is confusing (and not checkable with static typing, leading
  to ugly debug code), and hostPorts are not provided by Mesos.

- Replaced most hostPort-based data structures and fields as above.

- Simplified ClusterTaskSetManager to deal with preferred locations in a
  more concise way and generally be more concise.

- Updated the way ClusterTaskSetManager handles racks: instead of
  enqueueing a task to a separate queue for all the hosts in the rack,
  which would create lots of large queues, have one queue per rack name.

- Removed non-local fallback stuff in ClusterScheduler that tried to
  launch less-local tasks on a node once the local ones were all
  assigned. This change didn't work because many cluster schedulers send
  offers for just one node at a time (even the standalone and YARN ones
  do so as nodes join the cluster one by one). Thus, lots of non-local
  tasks would be assigned even though a node with locality for them
  would be able to receive tasks just a short time later.

- Renamed MapOutputTracker "generations" to "epochs".
More scheduling fixes:

- Added periodic revival of offers in StandaloneSchedulerBackend

- Replaced task scheduling aggression with multi-level delay scheduling
  in ClusterTaskSetManager

- Fixed ZippedRDD preferred locations because they can't currently be
  process-local

- Fixed some uses of hostPort
Comment cleanup (via Kay) and some debug messages
Address some review comments:

- When a resourceOffers() call has multiple offers, force the TaskSets
  to consider them in increasing order of locality levels so that they
  get a chance to launch stuff locally across all offers

- Simplify ClusterScheduler.prioritizeContainers

- Add docs on the new configuration options
Added some comments on threading in scheduler code
Added unit tests for ClusterTaskSetManager, and fix a bug found with
resetting locality level after a non-local launch
Allow subclasses of Product2 in all key-value related classes (ShuffleDependency, PairRDDFunctions, etc).
Added the missing RDD files and cleaned up SparkContext.
Made PairRDDFunctions taking only Tuple2, but made the rest of the shuffle code path working with general Product2.
Added a test for sorting using MutablePair's.
Code review feedback. (added tests for cogroup and substract; added more documentation on MutablePair)
Small fixes to web UI:

- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
Merge pull request #847 from rxin/rdd

Allow subclasses of Product2 in all key-value related classes