JuliaLang/julia — pull request #8306 by amitmurthy (2014-09-11 04:10:06+00:00)
Title: RFC: reworked cluster manager interface

Addresses issues raised in #8168

- Appropriate `launch` and `manage` methods are called depending on the cluster manager used
- `launch` - i.e., launch of julia process, `create_worker` - i.e., connect to the new worker and `add_worker` - i.e., setup connections to other workers are all done concurrently in separate tasks - addresses @simonster 's issue in #8168.
- `launch` is further broken up into separate tasks by host in the SSHManager
- Each of them respects the value of kw arg `max_parallel` i.e., the maximum number of concurrent launches active at a host. - addresses @floswald 's issue in #8168
- All of these tasks communicate with each other by means of a Queue and Condition variable. Native support for a `Channel` would have been useful here, but that is a separate PR.
- NOTE: The flags `-o ControlMaster auto -o ControlPath [path] -o ControlPersist 5` suggested in #7589 seem to only reuse the socket connection. The authentication for each ssh session still happens independently and we still hit `MaxStartups` even if the afore mentioned flags are present.
- breaks the current ClusterManager.jl
- changes to ClusterManager.jl, doc updates and tests will be added once we agree on this interface