JuliaLang/julia — pull request #4580 by timholy (2013-10-19 18:25:34+00:00)
Title: RFC: Shared-memory parallel computing

This is 0.3 material or beyond.

A recent mailing-list [discussion](https://groups.google.com/d/msg/julia-dev/vYtrPgjQl8o/WsQlJ-XUlyYJ), and particularly a comment by Jason Riedy, pointed out a potential strategy for shared-memory parallelism that circumvents some of the problems with other strategies. I've fleshed that basic idea out here, hopefully in a way that provides a fairly nice Julian interface.

This is an RFC for several important reasons: at the moment it's Linux-only, there are API issues to consider, I haven't integrated it into the build process of Julia (it's easier to test and tweak this way, if someone else wants to do so), there are no docs, etc.

Noteworthy features:
- This presents both a natural "array" interface and something suitable for parallelism, which I find to be the most attractive thing about this implementation. The main trick here is subverting the serializer so that we can have the full array version available in process id 1 without a big transport hit (the other processes just see the remote-ref version).
- Compared to PTools' `pfork`, this has a more modest overhead. On my machine `pfork` has approximately 500ms minimum latency even for a trivial computation. Here, the `pcall` form is limited by the event queue, and on my machine seems to have a minimum latency of about 40ms. The busy-wait version `pcall_bw` gets that down to less than 1ms per process.
- It uses shared memory but cleans up after itself nicely, so is a little like Amit's shared memory/server tasks without the risk of leaving resources dangling.

In my view the main API issue to consider is how to handle the return value(s). As an example of the conceptual challenge, many Julia functions have a return like `fill!()`, where the filled array is returned---this is nice because then you can chain together function calls very naturally. However, serializing that output will result in a huge performance hit. So the convention here is to return two things: the full output of the function call on process id 1 (which requires no serialization), and the output of each other process as a remote reference. In cases where the output is already being stored in a pre-allocated `SharedArray`, none of these outputs matter anyway, because such results are immediately available in the array that exists in process id 1.

CC @amitmurthy.