JuliaLang/julia — pull request #9434 by amitmurthy (2014-12-21 16:25:21+00:00)
Title: RFC: user defined transports

This builds on #9309 and supports user definable transports.
It supersedes #9046 - the discussion there outlines much of the motivations for this PR.

To implement this, custom cluster managers would need to provide their own
`connect(manager::FooManager, pid::Integer, config::WorkerConfig)` method

`connect` should return a pair of `AsyncStream` objects, one for reading data sent from worker `pid`, and the other to write data that needs to be sent to worker `pid`. custom cluster managers can use an in-memory `BufferStream` as the plumbing to ferry data between the custom, non AsyncStream transport and the Julia parallel infrastructure.

A `BufferStream` wraps a `PipeBuffer` and condition variables to make a waitable stream.

`examples/clustermanager/0mq` is an example of how they are used to setup a star network with a 0MQ broker in the middle.

`connect` is optional, and the default implementation is based on TCP as a transport mechanism.

Another optional method is `kill(manager::FooManager, pid::Int, config::WorkerConfig)`, which is called to remove a worker from the cluster.

Two example implementations are provided:
- `examples/clustermanager/simple` shows the use of unix domain sockets as transport
- `examples/clustermanager/0mq` shows the use of 0MQ as transport

One thought is to move the examples to package `ClusterManagers.jl` instead of adding it here. Have kept it in this PR so that folks can have a look. Will move it based on feedback.

Documentation still needs to updated.