JuliaLang/julia — pull request #3428 by amitmurthy (2013-06-17 19:33:45+00:00)
Title: RFC: detach() and attach() from/to worker processes

This patch is built on changes in open PR https://github.com/JuliaLang/julia/pull/3394 .

It provides a means for the client REPL to be detached from the set of worker processes and reattached later. Typical uses cases:

- Long running computation in julia - spanning hours/days. The client terminal can be safely detached after starting the computation
- Similar to above, when the workers are in a cloud environment but the client console is the users local machine/laptop.

Currently implemented are ```detach``` and ```attach```. Both can only be executed on the client (id =1) process.

```detach(connection_file::String)``` safely removes the client from the process group and writes complete connection information to the ```connection_file```.

```attach(connection_file::String)``` uses the information as garnered during detach and reconnects to the cluster.

Only one client process can be connected to the cluster at any time.

TO BE DONE
--------------------

Since the above results in the client (id = 1) being detached, it leads to certain issues with the parallel computing infrastructure we currently have - for example, @parallel and pmap. In typical usage the client process (id =1) will be the controller processes for the entire distributed job execution and currently is meant to be interactive - in the sense the terminal (REPL) is expected to be kept open.

Since the client process can now be detached and closed, we should provide an alternative mechanism for the user to push computation work to the "background", query its status and retrieve results independent of the client process.

We can have the following new macros/functions:

```@bg_exec(key::String, code_block)``` - runs the block of code, code_block, in the background, i.e., on the worker with the lowest process id, (lowest, non pid=1 process). The result of the block of code will be stored in a Dict on the said worker with the specified ```key```

```bg_clear()``` - clears the dict on the worker where background jobs are controlled from.

```bg_take(key), bg_fetch(key)``` - takes and fetches the responses from the dict

```bg_put(key)``` - application code can add its own information to this Dict, for example progress information on long running computations that can be queried periodically.

NOTE:

- The above scheme puts the process with lowest pid (other than the client) in a special role of keeping some state of the long running computational tasks.

- given the co-operative multitasking nature of Julia, application code must play nice and periodically call yield() in order for the client process to be able to attach() and detach() at will.

- Due to the seg fault mentioned in PR # 3394, extensive testing of detach/attach has not been possible yet.

Any issues/ suggestions / different schemes for having a clean and consistent implementation of ```client detach/attach``` is welcome.