mesos/spark — pull request #602 by rxin (2013-05-08 05:45:43+00:00)
Title: Manual merge & cleanup of Shane's Shuffle Performance Optimization

Merged #557 (originally #522) to master branch.

Did some clean ups too. Note that most of the classes in spark.network are moved into their own files without any changes.

Quoting #557:

The patch includes below optimizations for shuffle performance.
1. Use netty embeded OIO file server instead of Connection Manager for block shuffling.
2. Skip 0-byte block requests to reduce the network transport in shuffle

A system property "spark.shuffle.use.netty" is provided for user to choose whether enable or disable this optimization. If spark.shuffle.use.netty=true, the optimization is enabled, otherwise it is disabled. By default it is disabled.