JuliaLang/julia — pull request #5775 by timholy (2014-02-12 11:56:36+00:00)
Title: RFC: allowing driver and worker processes to load different modules

On a single machine, I have a driver process that needs to interact with the user; it makes use of several graphics/plotting packages & modules. This driver process delegates its computational work to several worker processes, each of which relies on quite a large number of compute-focused packages & modules. Loading all this code into a single process takes approximately 1 minute, which is a large barrier to entry. But in principle, there's no reason the driver process needs to load the compute packages, and there's no reason the worker processes need to load the graphics packages. So it seems this time could be shaved down quite a bit by some partitioning.

I did some experiments trying to get this working, and with current `master` the strategy I found seems a bit sub-optimal (this PR may improve matters). The main challenge is that the driver needs to call functions inside modules that it knows nothing about. C permits this by allowing you to declare a function without defining it, but I'm not aware of anything similar in Julia. So the solution seems to be to rely on `eval` inside of `Main`.

With current `master`, here are the various files I needed:
```
module MyWorker

export compute

compute(x) = x+1

end
```
```
module Glue

export evalfetch

evalfetch(r::RemoteRef) = eval(Main, fetch(r))

end
```
```
module MyDriver

using Glue

export main

function main(wpid)
# Load the code that the worker process needs to do its job.
# It needs to be evaled in Main to prevent the serializer from
# complaining about not knowing about MyDriver on wpid
# (this is not an issue if you execute this from the REPL)
wait(eval(Main, :(@spawnat $wpid include("MyWorker.jl"))))

# Now set up the function call, but wrap it in an expression to avoid
# errors due to not knowing about MyWorker
rr = RemoteRef(wpid)
x = 15
put(rr, :(MyWorker.compute($x)))
return remotecall_fetch(wpid, evalfetch, rr)
end

end
```
Then run this as follows (I'm using `include` to avoid any possible funny business with `require` loading things on all workers):
```
wpid = addprocs(1)[1]

@everywhere using Glue

include("MyDriver")
MyDriver.main(wpid)
```
and you get back `16`.

In an ideal world, I'd even put these first two lines inside `MyDriver.main()`; it seems better not to have to bother the user with spawning new processes and loading code on them. But it seems hard to get Glue into MyDriver after-the-fact.

With this PR, in addition to `MyWorker.jl`, here's what you need:

```
module SimpleDriver

export main

function main()
wpid = addprocs(1)[1]
wait(eval(Main, :(@spawnat $wpid include("MyWorker.jl"))))

remotecall_fetch(wpid, :(MyWorker.compute), 15)
end

end
```

```
include("SimpleDriver.jl")
SimpleDriver.main()
```

I don't (yet!) have a good understanding of how `multi.jl` works, so this should be viewed as a first attempt.