apache/spark — pull request #3465 by aarondav (2014-11-26 02:48:32+00:00)
Title: [SPARK-4516] Avoid allocating unnecessarily Netty PooledByteBufAllocators

Turns out we are allocating an allocator pool for every TransportClient (which means that the number increases with the number of nodes in the cluster), when really we should just reuse one for all clients.

This patch, as expected, greatly decreases off-heap memory allocation, and appears to make allocation only proportional to the number of cores.