dmtcp/dmtcp — pull request #13 by artpol84 (2014-10-22 12:10:54+00:00)
Title: [RM plugin]

My comments:
This patch fixes hang of checkpoint process and works correctly on my laptop slurm installation. I still have problems at Ohio but they seems to be related to the different issue - UIDs are duplicated like this:
ckpt_binary_3f11e67a5b8644bc-43000-5447894f.dmtcp
ckpt_binary_eacc16b19f07e1d-45000-5447894f.dmtcp
ckpt_dmtcp_srun_helper_3f11e67a5b8644bc-40000-5447894f.dmtcp
ckpt_hydra_pmi_proxy_3f11e67a5b8644bc-43000-5447894f.dmtcp
ckpt_hydra_pmi_proxy_eacc16b19f07e1d-45000-5447894f.dmtcp
ckpt_mpiexec.hydra_3f11e67a5b8644bc-40000-5447894f.dmtcp
-------------------------------------------------------------8<---------------------------------------------------------------------------

Original patch message:

OMPI doesn't use srun's IO manager but hydra does.
So at checkpoint time DMTCP tries to process fd's connected to srun and this hangs everything.

For that purposes dmtcp_srun_helper was introduced. The idea was derived from Kapil's SSH plugin
but no fd passing implemented now. I found easier solution and hope that it will work in all
circumstances:
at initial run hydra calls dmtcp_srun_helper and it runs srun and all other processes
(so initially hydra and dmtcp_srun helper are not under srun control)
at restart time we run everything under srun so all processes are connected to srun and their output will
go to the new slurm batch output which is very convinient since we don't need to fixs hydra stdio. It stops
receive any output and all output goes directly to the output file.

This cheme may be incomplete and have some drawbacks. More sophisticated approach would be to run hydra and helper
outside of srun but this will be slightly more complicated:
1. Will need to detect hydra and helper checkpoints and process them differently.
2. Will need to fix hydra IO to the new output.
While current solution works we will keep it. Once it won't be enough we will switch to this more complete scheme described.