Pyotrch distributed: Running shell command

shaibagon: <p>I’m running a <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">distributed pytorch training</a>. Everything works like charm. I am fully utilizing all GPUs, all processes are in sync, everything is fine.<br>
At the end of each epoch, I want to run some elaborate evaluation in a <em>new</em> process (not to block the training):</p>
<pre><code class="lang-python">if args.rank == 0:
  # only for the "main" rank
  subprocess.run(['python3', 'my_eval_code.py', '--chk', 'checkpoint'])
</code></pre>
<p>At this point, execution stops, the new process is <em>not</em> started and everything just halts.</p>
<ol>
<li>Is there some interdependence between pytorch’s DDP and <code>subprocess</code> module?</li>
<li>How can I start a new shell script (<code>subprocess.run</code>/<code>subprocess.call</code>/<code>subprocess.popen</code>) from inside a <code>DDP</code> process?</li>
</ol>
<p>I also posted this question on <a href="https://stackoverflow.com/q/68516184/1714410" rel="noopener nofollow ugc">SO</a>.</p>
<hr>
<p><strong>Update (July 29th, 2021)</strong><br>
I changed my code to:</p>
<pre><code class="lang-auto">proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
print(f'\t{proc}={proc.poll()}')
try:
  proc_o, proc_e = proc.communicate(timeout=120)
  print(f'successfully communicated o={proc_o} e={proc_e} poll={proc.poll()}')
except subprocess.TimeoutExpired:
  proc.kill()
  proc_o, proc_e = proc.communicate()
  print(f'time out o={proc_o} e={proc_e} poll={proc.poll()}')
</code></pre>
<p>No good: the <code>Popen</code> command is blocking, the print of the <code>poll</code> command is never executed, let alone the <code>communicate</code>.<br>
When I check on the job with <code>top</code>, I see:</p>
<pre><code class="lang-auto">  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
37924 bagon     39  19   23640   2796    880 S  15.8  0.1   0:15.34 python3
</code></pre>
<p>Looking at the process that actually runs: I see this:</p>
<pre><code class="lang-auto">UID        PID  PPID  C STIME TTY      STAT   TIME CMD
bagon    37924 37065  1 08:00 ?        SNl    0:15 /home/bagon/.conda/envs/my_env/bin/python3 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=50, pipe_handle=54) --multiprocessing-fork
</code></pre>
<p>It seems like there is some underlying mechanism preventing <code>subprocess</code> module from starting new processes.</p>
<p>Any help?</p>

wanchaol: <p>Thanks for posting <a class="mention" href="/u/shaibagon">@shaibagon</a>. For your questions:<br>
1.Yeah I think there’re some connection because PyTorch using multiprocessing and it has some interdependencies with subprocess<br>
2. I think if you don’t want to block training, did you try if <code>subprocess.popen</code> works? could you also try multiprocessing.Process and see if that works?</p>

shaibagon: <p>I tried <code>subprocess.popen</code> as well I did not see any process starts and it is unclear what exactly is the problem - I don’t see any errors/feedback only that the execution halts.</p>

shaibagon: <p>I opened a <a href="https://github.com/pytorch/pytorch/issues/62381" rel="noopener nofollow ugc">bug report</a> describing this issue along with a minimal script that preproduces this deadlock.</p>
<p>Fingers crossed.</p>

H-Huang: <p>I answered your issue on github <a href="https://github.com/pytorch/pytorch/issues/62381" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed and subprocess do not work together? · Issue #62381 · pytorch/pytorch · GitHub</a>. Hopefully that clarifies how to use <code>init_process_group</code>. But I suspect this may not address the original issue brought up in this post? It is hard to say why <code>my_eval_code.py</code> is blocking and what exactly that code is doing. If you can find a locally reproducible example where it blocks that would help!</p>