Torch.multiprocess.spawn with join=False gives the following error

username: ayushm-agrawal
cooked: <p>Hello all,<br>
I am trying to implement Distributed Parallel for my model and I followed the ImageNet example for this. I am pretty new to distributed programming, so I am not sure about a bunch of things. When I use torch.multiprocess.spawn with join=True, there is no output that is printed. When I change the join to False, I get the following error below.</p>
<pre><code class="lang-auto">&lt;torch.multiprocessing.spawn.SpawnContext object at 0x2b49eee8a3c8&gt;
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
</code></pre>
<p>I am submitting this job through a SLURM script that I have put below as well.</p>
<pre><code class="lang-auto">#!/bin/sh
#SBATCH --ntasks=4
#SBATCH --time=60:00:00
#SBATCH --partition=gpu
#SBATCH --mem=64gb
#SBATCH --nodes=2
#SBATCH --gres=gpu:2
#SBATCH --constraint=gpu_32gb
#SBATCH --job-name=test
#SBATCH --output=.../out_files/test.out

export PYTHONPATH=$WORK/tf-gpu-pkgs
module load singularity
singularity exec docker://&lt;user&gt;/pytorch-opencv:latest	python3 -u $@ --use_adam=1 --multiprocessing_distributed --benchmarks=0 --benchmark_arch='vgg19' --batch_size=128 --test=0 --transfer=0 --dataset='&lt;dataset-here&gt;'
</code></pre>
<p>My code is like the ImageNet example and I am not sure what I am doing wrong.</p>
<p>Thank you,</p>

username: osalpekar
cooked: <p>Are you capturing the <code>SpawnContext</code> object returned by the call to <code>torch.multiprocess.spawn</code>? This <code>SpawnContext</code> is returned only when <code>join=False</code>, and must be saved for the spawned processes to coordinate IPC. If you allow the object to be destructed, you will see this error.</p>
<p>Here is a GitHub issue with some more information: <a href="https://github.com/pytorch/pytorch/issues/30461" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/30461</a></p>

username: ayushm-agrawal
cooked: <p>Hello Omkar,<br>
Thank you for replying. The weird issue is that I don’t see the terminated print statement when I use join=True. With the issue that you linked to me, when I spawn the process, shouldn’t I be seeing the print statements from my <strong>main_worker</strong> function before I hit the terminated print statement? I apologize if this question isn’t framed right. I am new to distributed and don’t understand the system that well.</p>
<pre><code class="lang-auto"> ctx = mp.spawn(main_worker, nprocs=ngpus_per_node,
                 args=(ngpus_per_node, args), join=False)
        time.sleep(3)
        print('terminated')
        ctx.join()
    else:
        # Simply call main_worker function
        main_worker(args.gpu, ngpus_per_node, args)


def main_worker(gpu, ngpus_per_node, args):
    global best_acc1
    print(gpu)
    args.gpu = gpu
</code></pre>
<pre><code class="lang-auto">diff --git a/repro_org.py b/repro.py
index be44c3d..e971db4 100644
--- a/repro_org.py
+++ b/repro.py
@@ -6,7 +6,8 @@ def worker(nproc, arg1, arg2, arg3):
     test = True
 
 if __name__ == '__main__':
-    mp.spawn(worker, (None, None, None), nprocs=1, join=False)
+    ctx = mp.spawn(worker, (None, None, None), nprocs=1, join=False)
     time.sleep(3)
     print('terminated')
+    ctx.join()
</code></pre>