apache/spark — pull request #2259 by davies (2014-09-04 01:13:46+00:00)
Title: [SPARK-3030] [PySpark] Reuse Python worker

Reuse Python worker to avoid the overhead of fork() Python process for each tasks. It also tracks the broadcasts for each worker, avoid sending repeated broadcasts.

This can reduce the time for dummy task from 22ms to 13ms (-40%).

It's enabled by default, could use `spark.python.worker.reuse = false` to disable it.