apache/spark — pull request #3417 by davies (2014-11-22 20:04:23+00:00)
Title: [SPARK-4548] improve performance of python broadcast

Re-implement the Python broadcast using file:

1) serialize the python object using cPickle, write into disks.
2) Create a wrapper in JVM (for the dumped file), it read data from during serialization
3) Using TorrentBroadcast or HttpBroadcast to transfer the data (compressed) into executors
4) During deserialization, writing the data into disk.
5) Passing the path into Python worker, read data from disk and unpickle it into python object, until the first access.

It fixes the performance regression introduced in #2659, has similar performance as 1.1, but support object larger than 2G, also improve the memory efficiency (only one compressed copy in driver and executor).

Testing with a 500M broadcast and 4 tasks (excluding the benefit from reused worker in 1.2):
name | 1.1 | 1.2 with this patch |
==============|========|========|======
python-broadcast-w-bytes |	25.20 |	9.33 |	170.13% |
python-broadcast-w-set	| 4.13	| 4.50 |	-8.35% |