Problem in DataPrallel for more than 2 GPUs

username: Ben_Marinberg
cooked: <p>I had a working model. It ran and was very consistent with its result regarding the number of GPUs.<br>
This means it already gave similar results when running on 1-4 GPUs (all combinations had been tested).<br>
Now a came across some weird issue, Everything works fine on single or double GPUs (done with ““model = nn.DataParallel(model, device_ids=devices)””).<br>
If I try to use 3 GPUs it crashes with  NCCL Error 2: unhandled system error.<br>
Moreover, if I use 4 GPUs it just gets stuck, ctrl+c doesn’t help of course (luckily I’m working via terminal and ssh so I just break the pipe to kill the run).</p>

username: ptrblck
cooked: <p>Could you rerun the 3 GPU run via <code>NCCL_DEBUG=INFO python scripy.py args</code> and see, if NCCL reports any errors?</p>

username: Ben_Marinberg
cooked: <p>The first row is my print command</p>
<blockquote>
<p>Initiating epoch 0<br>
5f8f287430a2:27239:27239 [0] NCCL INFO Bootstrap : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
5f8f287430a2:27239:27239 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>5f8f287430a2:27239:27239 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
5f8f287430a2:27239:27239 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
5f8f287430a2:27239:27239 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda10.2<br>
5f8f287430a2:27239:27499 [0] NCCL INFO Channel 00/02 :    0   1   2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64<br>
5f8f287430a2:27239:27501 [2] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64<br>
5f8f287430a2:27239:27499 [0] NCCL INFO Channel 01/02 :    0   1   2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1 [1] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1<br>
5f8f287430a2:27239:27501 [2] NCCL INFO Trees [0] -1/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;-1/-1/-1 [1] -1/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;-1/-1/-1<br>
5f8f287430a2:27239:27500 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff<br>
5f8f287430a2:27239:27501 [2] NCCL INFO Setting affinity for GPU 3 to ffff0000,ffff0000<br>
5f8f287430a2:27239:27499 [0] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64<br>
5f8f287430a2:27239:27499 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [1] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1<br>
5f8f287430a2:27239:27499 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff<br>
5f8f287430a2:27239:27500 [1] NCCL INFO Could not enable P2P between dev 1(=5e000) and dev 0(=3b000)</p>
<p>5f8f287430a2:27239:27499 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device</p>
<p>5f8f287430a2:27239:27500 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device</p>
<p>5f8f287430a2:27239:27501 [2] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device<br>
5f8f287430a2:27239:27499 [0] NCCL INFO include/shm.h:41 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO include/shm.h:41 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO include/shm.h:41 → 2</p>
<p>5f8f287430a2:27239:27499 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-2-0 (size 9637888)</p>
<p>5f8f287430a2:27239:27500 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-0-1 (size 9637888)</p>
<p>5f8f287430a2:27239:27499 [0] NCCL INFO transport/shm.cc:101 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO transport/shm.cc:101 → 2</p>
<p>5f8f287430a2:27239:27501 [2] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-1-2 (size 9637888)</p>
<p>5f8f287430a2:27239:27499 [0] NCCL INFO transport.cc:30 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO transport/shm.cc:101 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO transport.cc:30 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO transport.cc:30 → 2<br>
5f8f287430a2:27239:27499 [0] NCCL INFO transport.cc:49 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO transport.cc:49 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO transport.cc:49 → 2<br>
5f8f287430a2:27239:27499 [0] NCCL INFO init.cc:766 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO init.cc:766 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO init.cc:766 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO init.cc:840 → 2<br>
5f8f287430a2:27239:27500 [1] NCCL INFO init.cc:840 → 2<br>
5f8f287430a2:27239:27499 [0] NCCL INFO init.cc:840 → 2<br>
5f8f287430a2:27239:27501 [2] NCCL INFO group.cc:73 → 2 [Async thread]<br>
5f8f287430a2:27239:27500 [1] NCCL INFO group.cc:73 → 2 [Async thread]<br>
5f8f287430a2:27239:27499 [0] NCCL INFO group.cc:73 → 2 [Async thread]<br>
5f8f287430a2:27239:27239 [0] NCCL INFO init.cc:906 → 2<br>
Traceback (most recent call last):<br>
File “main_comb.py”, line 75, in <br>
[trloss,tstloss,tstevm,inf_time]=cnn.run_epoch(epoch,show=True)<br>
File “/common_space_docker/storage_4TSSD/ben/ce/Results/Nov_19_20_11:02_3DFF_base/cnn_imp.py”, line 875, in run_epoch<br>
train_loss = train_model()<br>
File “/common_space_docker/storage_4TSSD/ben/ce/Results/Nov_19_20_11:02_3DFF_base/cnn_imp.py”, line 762, in train_model<br>
outputs = model(channel)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py”, line 160, in forward<br>
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py”, line 165, in replicate<br>
return replicate(module, device_ids, not torch.is_grad_enabled())<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py”, line 88, in replicate<br>
param_copies = _broadcast_coalesced_reshape(params, devices, detach)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py”, line 71, in _broadcast_coalesced_reshape<br>
tensor_copies = Broadcast.apply(devices, *tensors)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/_functions.py”, line 22, in forward<br>
outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/comm.py”, line 56, in broadcast_coalesced<br>
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)<br>
RuntimeError: NCCL Error 2: unhandled system error</p>
</blockquote>
<p>These are 3 TITAN GPUs with more than enough space (a batch and the model is about 4Gb of 25Gb available on each TITAN)</p>

username: ptrblck
cooked: <p>The error seems to come from NCCL not being able to use shared memory:</p>
<pre><code class="lang-python">NCCL WARN Call to posix_fallocate failed : No space left on device
[...]
NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-2-0 (size 9637888)
</code></pre>
<p>so make sure your machine has enough shared memory and if you are using a docker container, use the <code>--ipc=host</code> argument (or specify the shared memory usage manually).</p>

username: Ben_Marinberg
cooked: <p>I am using a docker container on a server, though the server has 750Gb RAM. It means that there is over 500Gb free after loading the datasets.<br>
via nvidia-smi, I can see that all GPUs get the data and it fails as soon as the last GPU gets its batch.<br>
Also, I usually see that after dataparallel all GPUs get a replica of the model while currently, only the master GPU gets the model and the rest get it together with the batch (if it helps).</p>

username: ptrblck
cooked: <p>The error is raised by insufficient shared memory not system RAM. Since you are using a container, did you try the suggested argument?</p>

username: Ben_Marinberg
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="103373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>insufficient shared memory</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> you’re some kind of a magician!<br>
–ipc flag worked it out, everything works with 1-4 devices!</p>

username: emmb
cooked: <p>I am having a very similar issue in using nn.DataParallel. I am working on 3 Tesla V100 GPUs. When I use two GPUs the code runs perfectly. But when I use all three GPUs the code fails during the forward pass. There is no error information given. The only error message I get is “Killed”. Could you help me with one, please? I am not using any docker.</p>

username: ptrblck
cooked: <p><code>Killed</code> usually indicates that the OS has killed the process e.g. due to insufficient host memory (RAM).<br>
Could you check, if this could be the case for the 3GPU run, e.g. are you loading more data into the RAM in this use case?</p>

username: emmb
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Actually I have a lot RAM in excess (more than 300 GB). So, that cannot be the case. Also, I tried batch size=1, still didn’t work for the 3 GPU run.</p>
<p>Another thing I noticed: when I set num_workers=0 in DataLoader, it works perfectly for the 3 GPU run. But setting num_workers=0 increases the training time drastically so I don’t like it. Is there an alternate fix (i.e. to use 3 GPUs while keeping my num_workers&gt;0)?</p>

username: ptrblck
cooked: <p>OK, in that case try to get a stacktrace by running the script in <code>gdb</code> via:</p>
<pre><code class="lang-auto">gdb --args python script.py args
...
run
...
bt
</code></pre>
<p>which should hopefully show what’s failing.</p>

username: emmb
cooked: <p>I did. It is actually “RuntimeError: DataLoader worker (pid 27772) is killed by signal: Killed.” Any fix for this?</p>

username: emmb
cooked: <p>Switching to pytorch 1.7.0 solved the issue.</p>