Squashed commit of the following:

commit f980ab025dc3ba54fe5ea7aaacdb67382ba16dd5
Author: Aaron Davidson <aaron@databricks.com>
Date:   Thu Oct 16 20:20:05 2014 -0700

    Fix Nio compile issue

commit 939f276fe42feb3c333e39f371e9c6400fe22ddc
Author: Aaron Davidson <aaron@databricks.com>
Date:   Sun Oct 12 16:45:55 2014 -0700

    Attempt to make comm. bidirectional

commit 29c6dcfaacb2e8b1f0582c6d5e435349c52e29af
Author: Aaron Davidson <aaron@databricks.com>
Date:   Sun Oct 5 17:58:43 2014 -0700

    [SPARK-3453] Netty-based BlockTransferService, extracted from Spark core

    This PR encapsulates #2330, which is itself a continuation of #2240. The first goal of this
    PR is to provide an alternate, simpler implementation of the ConnectionManager which is based on Netty.

    In addition to this goal, however, we want to resolve [SPARK-3796](https://issues.apache.org/jira/browse/SPARK-3796), which calls for a
    standalone shuffle service which can be integrated into the YARN NodeManager, Standalone Worker, or
    on its own. This PR makes the first step in this direction by ensuring that the actual Netty service
    is as small as possible and extracted from Spark core. Given this, we should be able to construct
    this standalone jar which can be included in other JVMs without incurring significant dependency or
    runtime issues. The actual work to ensure that such a standalone shuffle service would work in Spark
    will be left for a future PR, however.

    In order to minimize dependencies and allow for the service to be long-running (possibly
    much longer-running than Spark, and possibly having to support multiple version of Spark
    simultaneously), the entire service has been ported to Java, where we have full control
    over the binary compatibility of the components and do not depend on the Scala runtime or
    version.

    These PRs have been addressed by folding in #2330:

    SPARK-3453: Refactor Netty module to use BlockTransferService interface
    SPARK-3018: Release all buffers upon task completion/failure
    SPARK-3002: Create a connection pool and reuse clients across different threads
    SPARK-3017: Integration tests and unit tests for connection failures
    SPARK-3049: Make sure client doesn't block when server/connection has error(s)
    SPARK-3502: SO_RCVBUF and SO_SNDBUF should be bootstrap childOption, not option
    SPARK-3503: Disable thread local cache in PooledByteBufAllocator

    TODO before mergeable:
    [ ] Implement uploadBlock()
    [ ] Unit tests for RPC side of code
    [ ] Performance testing
    [ ] Turn OFF by default (currently on for unit testing)

commit f7e7568414692989215d97abce9dda2fe172abb4
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 30 12:28:21 2014 -0700

    Fixed spark.shuffle.io.receiveBuffer setting.

commit 5d98ce3de1deeeb7fbdc26b9303a591c46f1892b
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 30 00:56:32 2014 -0700

    Flip buffer.

commit f6c220df8406be14fbdb7270682727e1085518a4
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 23:30:17 2014 -0700

    Merge with latest master.

commit 407e59afd3cb7385af9f63dc2263a40c7c21d783
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 19:37:28 2014 -0700

    Fix style violation.

commit a0518c766f0f4eba24459ffac61dce789fc14092
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 19:22:34 2014 -0700

    Implemented block uploads.

commit 4b18db29edcdb87577fd033835275fd1c2957dcd
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 15:45:05 2014 -0700

    Copy the buffer in fetchBlockSync.

commit bec4ea2b54659cfed6f54e527aa878dfbff829c7
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 12:22:01 2014 -0700

    Removed OIO and added num threads settings.

commit 1bdd7eec5d9ddb5a9eb33c9733878aea3ca26ba6
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 12:07:53 2014 -0700

    Fixed tests.

commit d68f3286a4a9795dfb61a8a63b8a20b3eafb4821
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 11:30:13 2014 -0700

    Logging close() in case close() fails.

commit f63fb4c1976e503238b7d7151f8f45f40ced36e9
Author: Reynold Xin <rxin@apache.org>
Date:   Mon Sep 29 11:13:44 2014 -0700

    Add more debug message.

commit 6afc435037a0448d6eb243bd18411ef25e3a2cf7
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 16 22:51:11 2014 -0700

    Added logging.

commit c066309afbb0e248a8b2b808d997e6b37a2bff1e
Author: Reynold Xin <rxin@apache.org>
Date:   Fri Sep 12 22:42:32 2014 -0700

    Implement java.io.Closeable interface.

commit 519d64dcb7768b3657438a4cfc85ee8065f56c2a
Author: Reynold Xin <rxin@apache.org>
Date:   Fri Sep 12 14:18:58 2014 -0700

    Mark private package visibility and MimaExcludes.

commit f0a16e9ec7d5c811dff3cd5219548e05077099c8
Author: Reynold Xin <rxin@apache.org>
Date:   Fri Sep 12 00:40:53 2014 -0700

    Fixed test hanging.

commit 14323a55ebfa7ccc684c2ae78eac299a4426b353
Author: Reynold Xin <rxin@apache.org>
Date:   Thu Sep 11 22:13:02 2014 -0700

    Removed BlockManager.getLocalShuffleFromDisk.

commit b2f3281d0de540d38ea5b4c7bf576b775405d56d
Author: Reynold Xin <rxin@apache.org>
Date:   Thu Sep 11 22:12:08 2014 -0700

    Added connection pooling.

commit d23ed7bfd912770ace7eed7cd0dff2db6ac826e3
Author: Reynold Xin <rxin@apache.org>
Date:   Thu Sep 11 18:28:45 2014 -0700

    Incorporated feedback from Norman:
    - use same pool for boss and worker
    - remove ioratio
    - disable caching of byte buf allocator
    - childoption sendbuf/receivebuf
    - fire exception through pipeline

    In addition:
    - fire failure handler BlockFetchingListener at least once per block.
    - enabled a bunch of ignored tests

commit 9e0cb8736be6d38e3f30766271d28875ceca1ae8
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 21:04:56 2014 -0700

    Fixed BlockClientHandlerSuite

commit 5cd33d7798ae742e76107bb976d8478ab9476ae7
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 19:55:54 2014 -0700

    Fixed style violation.

commit cb589ec7b6d3758498249b63b395634efb83d8ba
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 19:01:23 2014 -0700

    Added more test cases covering cleanup when fault happens in ShuffleBlockFetcherIteratorSuite

commit 1be4e8ee7d932821c789cb974310e5d59df4ff84
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 01:11:40 2014 -0700

    Shorten NioManagedBuffer and NettyManagedBuffer class names.

commit 108c9edaed06c5e046a21c9a8e54c50390da9a0b
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 01:10:04 2014 -0700

    Forgot to add TestSerializer to the commit list.

commit b5c8d1fca6d3cf5c2b95395310200c8149a7eb16
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Sep 10 01:09:44 2014 -0700

    Fixed ShuffleBlockFetcherIteratorSuite.

commit 064747b50a591acb132b2c750957e79f54dfa88f
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 9 23:38:38 2014 -0700

    Reference count buffers and clean them up properly.

commit 2b44cf1b7547919bbe7386e954fe2f56be046790
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 9 14:36:31 2014 -0700

    Added more documentation.

commit 1760d3292ecf262e4c77c9e3b28bfd2900d25840
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 9 00:42:37 2014 -0700

    Use Epoll.isAvailable in BlockServer as well.

commit 165eab1518f5184ef9609f26d374c5ccefd05472
Author: Reynold Xin <rxin@apache.org>
Date:   Tue Sep 9 00:29:33 2014 -0700

    [SPARK-3453] Refactor Netty module to use BlockTransferService.

    Also includes some partial support for uploading blocks.