JuliaLang/julia — pull request #6855 by timholy (2014-05-15 15:36:31+00:00)
Title: Workers on same host: include code from filesystem, not node1

I've found that when using multiple processes, loading a large pile of code is particularly slow. This patch narrows the gap somewhat, for the case when all processes are on the same host. It works by avoiding interprocess communication and fetching the source files directly from the filesystem, presumably reducing the amount of time workers spend waiting for node 1 to get around to serving their requests.

Timing results:
For `using Distributions`:
Single-threaded (`julia`): ~15s
Multi-process, master (`julia -p 1`): ~24s
Multi-process, this patch (`julia -p 1`): ~20s

For `using Optim` (with its own internal `using Distributions` commented out):
Single-threaded (`julia`): ~4.2s
Multi-process, master (`julia -p 1`): ~7.3s
Multi-process, this patch (`julia -p 1`): ~5.7s

If anyone has suggestions to reduce the overhead even further, I'm all ears.