Error in my minimal working example of multiple GPUs?

username: peppermint
cooked: <p>I am trying to exploit multiple GPUs on Amazon AWS via DataParallel. This is on AWS Sagemaker with 4 GPUs, PyTorch 1.8 (GPU Optimized) and Python 3.6.</p>
<p>I have searched through the forum and read through <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">the data parallel tutorial</a> but I do not see such a minimal working example mentioned or the error explained.</p>
<p>Do you know what is wrong?</p>
<pre data-code-wrap="import"><code class="lang-import">x = torch.rand(300, 400, 500).cuda()

model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
model = torch.nn.DataParallel(model, device_ids=[0,1])
y = model(x)
</code></pre>
<p>I get an error:</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
4 model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))<br>
5 model = torch.nn.DataParallel(model, device_ids=[0,1])<br>
----&gt; 6 y = model(x)</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
916             result = self._slow_forward(*input, **kwargs)<br>
917         else:<br>
 → 918             result = self.forward(*input, **kwargs)<br>
919         for hook in itertools.chain(<br>
920                 _global_forward_hooks.values(),</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)<br>
153                 raise RuntimeError("module must have its parameters and buffers "<br>
154                                    "on device {} (device_ids[0]) but found one of "<br>
 → 155                                    “them on device: {}”.format(self.src_device_obj, t.device))<br>
156<br>
157         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</p>
<p>RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu</p>
</blockquote>
<p>For these variants, I get these errors:</p>
<pre><code class="lang-auto">import torch
x = torch.rand(300, 400, 500)

model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
model = torch.nn.DataParallel(model, device_ids=[0,1])
y = model(x)
```&gt; 
&gt; ---------------------------------------------------------------------------
&gt; RuntimeError                              Traceback (most recent call last)
&gt; &lt;ipython-input-133-2ed596eb6192&gt; in &lt;module&gt;
&gt;       4 model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
&gt;       5 model = torch.nn.DataParallel(model, device_ids=[0,1])
&gt; ----&gt; 6 y = model(x)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
&gt;     916             result = self._slow_forward(*input, **kwargs)
&gt;     917         else:
&gt; --&gt; 918             result = self.forward(*input, **kwargs)
&gt;     919         for hook in itertools.chain(
&gt;     920                 _global_forward_hooks.values(),
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
&gt;     153                 raise RuntimeError("module must have its parameters and buffers "
&gt;     154                                    "on device {} (device_ids[0]) but found one of "
&gt; --&gt; 155                                    "them on device: {}".format(self.src_device_obj, t.device))
&gt;     156 
&gt;     157         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
&gt; 
&gt; RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu

</code></pre>
<p>import torch<br>
x = torch.rand(300, 400, 500)</p>
<p>model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))<br>
model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()<br>
y = model(x)</p>
<pre><code class="lang-auto">&gt; ---------------------------------------------------------------------------
&gt; RuntimeError                              Traceback (most recent call last)
&gt; &lt;ipython-input-134-16dea105c595&gt; in &lt;module&gt;
&gt;       4 model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
&gt;       5 model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()
&gt; ----&gt; 6 y = model(x)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
&gt;     916             result = self._slow_forward(*input, **kwargs)
&gt;     917         else:
&gt; --&gt; 918             result = self.forward(*input, **kwargs)
&gt;     919         for hook in itertools.chain(
&gt;     920                 _global_forward_hooks.values(),
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
&gt;     164         if len(self.device_ids) == 1:
&gt;     165             return self.module(*inputs[0], **kwargs[0])
&gt; --&gt; 166         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
&gt;     167         outputs = self.parallel_apply(replicas, inputs, kwargs)
&gt;     168         return self.gather(outputs, self.output_device)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in replicate(self, module, device_ids)
&gt;     169 
&gt;     170     def replicate(self, module, device_ids):
&gt; --&gt; 171         return replicate(module, device_ids, not torch.is_grad_enabled())
&gt;     172 
&gt;     173     def scatter(self, inputs, kwargs, device_ids):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in replicate(network, devices, detach)
&gt;      89     params = list(network.parameters())
&gt;      90     param_indices = {param: idx for idx, param in enumerate(params)}
&gt; ---&gt; 91     param_copies = _broadcast_coalesced_reshape(params, devices, detach)
&gt;      92 
&gt;      93     buffers = list(network.buffers())
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in _broadcast_coalesced_reshape(tensors, devices, detach)
&gt;      69         # Use the autograd function to broadcast if not detach
&gt;      70         if len(tensors) &gt; 0:
&gt; ---&gt; 71             tensor_copies = Broadcast.apply(devices, *tensors)
&gt;      72             return [tensor_copies[i:i + len(tensors)]
&gt;      73                     for i in range(0, len(tensor_copies), len(tensors))]
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py in forward(ctx, target_gpus, *inputs)
&gt;      21         ctx.num_inputs = len(inputs)
&gt;      22         ctx.input_device = inputs[0].get_device()
&gt; ---&gt; 23         outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
&gt;      24         non_differentiables = []
&gt;      25         for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/comm.py in broadcast_coalesced(tensors, devices, buffer_size)
&gt;      56     devices = [_get_device_index(d) for d in devices]
&gt;      57     tensors = [_handle_complex(t) for t in tensors]
&gt; ---&gt; 58     return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
&gt;      59 
&gt;      60 
&gt; 
&gt; RuntimeError: NCCL Error 2: unhandled system error

</code></pre>
<p>import torch<br>
x = torch.rand(300, 400, 500).cuda()</p>
<p>model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))<br>
model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()<br>
y = model(x)</p>
<pre><code class="lang-auto">&gt; ---------------------------------------------------------------------------
&gt; RuntimeError                              Traceback (most recent call last)
&gt; &lt;ipython-input-136-439fd34aeaf9&gt; in &lt;module&gt;
&gt;       4 model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
&gt;       5 model = torch.nn.DataParallel(model, device_ids=[0,1]).cuda()
&gt; ----&gt; 6 y = model(x)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
&gt;     916             result = self._slow_forward(*input, **kwargs)
&gt;     917         else:
&gt; --&gt; 918             result = self.forward(*input, **kwargs)
&gt;     919         for hook in itertools.chain(
&gt;     920                 _global_forward_hooks.values(),
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
&gt;     164         if len(self.device_ids) == 1:
&gt;     165             return self.module(*inputs[0], **kwargs[0])
&gt; --&gt; 166         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
&gt;     167         outputs = self.parallel_apply(replicas, inputs, kwargs)
&gt;     168         return self.gather(outputs, self.output_device)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in replicate(self, module, device_ids)
&gt;     169 
&gt;     170     def replicate(self, module, device_ids):
&gt; --&gt; 171         return replicate(module, device_ids, not torch.is_grad_enabled())
&gt;     172 
&gt;     173     def scatter(self, inputs, kwargs, device_ids):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in replicate(network, devices, detach)
&gt;      89     params = list(network.parameters())
&gt;      90     param_indices = {param: idx for idx, param in enumerate(params)}
&gt; ---&gt; 91     param_copies = _broadcast_coalesced_reshape(params, devices, detach)
&gt;      92 
&gt;      93     buffers = list(network.buffers())
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in _broadcast_coalesced_reshape(tensors, devices, detach)
&gt;      69         # Use the autograd function to broadcast if not detach
&gt;      70         if len(tensors) &gt; 0:
&gt; ---&gt; 71             tensor_copies = Broadcast.apply(devices, *tensors)
&gt;      72             return [tensor_copies[i:i + len(tensors)]
&gt;      73                     for i in range(0, len(tensor_copies), len(tensors))]
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py in forward(ctx, target_gpus, *inputs)
&gt;      21         ctx.num_inputs = len(inputs)
&gt;      22         ctx.input_device = inputs[0].get_device()
&gt; ---&gt; 23         outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
&gt;      24         non_differentiables = []
&gt;      25         for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/comm.py in broadcast_coalesced(tensors, devices, buffer_size)
&gt;      56     devices = [_get_device_index(d) for d in devices]
&gt;      57     tensors = [_handle_complex(t) for t in tensors]
&gt; ---&gt; 58     return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
&gt;      59 
&gt;      60 
&gt; 
&gt; RuntimeError: NCCL Error 2: unhandled system error

</code></pre>
<p>import torch<br>
x = torch.rand(300, 400, 500).cuda()</p>
<p>model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))<br>
model = torch.nn.DataParallel(model.cuda(), device_ids=[0,1])<br>
y = model(x)</p>
<pre><code class="lang-auto">&gt; ---------------------------------------------------------------------------
&gt; RuntimeError                              Traceback (most recent call last)
&gt; &lt;ipython-input-137-a1695a6de8c1&gt; in &lt;module&gt;
&gt;       4 model = torch.nn.Sequential(torch.nn.Linear(500, 900), torch.nn.Linear(900, 1))
&gt;       5 model = torch.nn.DataParallel(model.cuda(), device_ids=[0,1])
&gt; ----&gt; 6 y = model(x)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
&gt;     916             result = self._slow_forward(*input, **kwargs)
&gt;     917         else:
&gt; --&gt; 918             result = self.forward(*input, **kwargs)
&gt;     919         for hook in itertools.chain(
&gt;     920                 _global_forward_hooks.values(),
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
&gt;     164         if len(self.device_ids) == 1:
&gt;     165             return self.module(*inputs[0], **kwargs[0])
&gt; --&gt; 166         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
&gt;     167         outputs = self.parallel_apply(replicas, inputs, kwargs)
&gt;     168         return self.gather(outputs, self.output_device)
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in replicate(self, module, device_ids)
&gt;     169 
&gt;     170     def replicate(self, module, device_ids):
&gt; --&gt; 171         return replicate(module, device_ids, not torch.is_grad_enabled())
&gt;     172 
&gt;     173     def scatter(self, inputs, kwargs, device_ids):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in replicate(network, devices, detach)
&gt;      89     params = list(network.parameters())
&gt;      90     param_indices = {param: idx for idx, param in enumerate(params)}
&gt; ---&gt; 91     param_copies = _broadcast_coalesced_reshape(params, devices, detach)
&gt;      92 
&gt;      93     buffers = list(network.buffers())
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py in _broadcast_coalesced_reshape(tensors, devices, detach)
&gt;      69         # Use the autograd function to broadcast if not detach
&gt;      70         if len(tensors) &gt; 0:
&gt; ---&gt; 71             tensor_copies = Broadcast.apply(devices, *tensors)
&gt;      72             return [tensor_copies[i:i + len(tensors)]
&gt;      73                     for i in range(0, len(tensor_copies), len(tensors))]
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py in forward(ctx, target_gpus, *inputs)
&gt;      21         ctx.num_inputs = len(inputs)
&gt;      22         ctx.input_device = inputs[0].get_device()
&gt; ---&gt; 23         outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
&gt;      24         non_differentiables = []
&gt;      25         for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]):
&gt; 
&gt; /opt/conda/lib/python3.6/site-packages/torch/nn/parallel/comm.py in broadcast_coalesced(tensors, devices, buffer_size)
&gt;      56     devices = [_get_device_index(d) for d in devices]
&gt;      57     tensors = [_handle_complex(t) for t in tensors]
&gt; ---&gt; 58     return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
&gt;      59 
&gt;      60 
&gt; 
&gt; RuntimeError: NCCL Error 2: unhandled system error</code></pre>

username: ptrblck
cooked: <p>Rerun your script with <code>NCCL_DEBUG=INFO python script.pt args</code> and check what causes the system error. The first error messages complaining about the device were caused as you didn’t push the parameters to the device via <code>.cuda()</code>.</p>

username: peppermint
cooked: <p>Thank you!</p>
<p>On torch 1.6, the following works:</p>
<pre><code class="lang-auto">#%env NCCL_DEBUG=INFO
import torch
a,b,c,d = 300,400,500,6000
x = torch.rand(a, b, c).cuda()

model = torch.nn.Sequential(torch.nn.Linear(c, d))
model = torch.nn.DataParallel(model.cuda(), device_ids=[0,1,2,3])

%timeit -n 5 y = model(x)
</code></pre>
<p>But on torch 1.8, Sagemaker prints the following warning and<br>
<code>INFO profiler_config_parser.py:102] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.</code><br>
and this error<br>
<code>RuntimeError: NCCL Error 2: unhandled system error</code></p>
<p>Do you have any idea what causes the trouble?</p>

username: ptrblck
cooked: <p>I guess setting the env variable inside the code didn’t work, as NCCL would output the entire setup of the node etc.</p>

username: peppermint
cooked: <p>Hi ptrblck,<br>
Here is the output with NCCL_DEBUG=INFO. Any ideas? This is on an AWS Sagemaker system with 4 GPUs. The code is</p>
<pre><code class="lang-auto">import torch
device = torch.device("cuda")

bs = 2**5
k = 2**10

data = torch.rand(bs, k).to(device)

model = torch.nn.Linear(k, k)
model = torch.nn.DataParallel(model)
model.to(device)

output = model(data)
</code></pre>
<blockquote>
<p>[2022-02-18 23:12:08.559 pytorch-1-8-gpu:844 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None<br>
[2022-02-18 23:12:08.588 pytorch-1-8-gpu:844 INFO profiler_config_parser.py:102] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.<br>
pytorch-1-8-gpu:844:844 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1&lt;0&gt; [1]veth-app1-2:169.255.254.2&lt;0&gt;</p>
<p>pytorch-1-8-gpu:844:844 [0] ofi_init:1136 NCCL WARN NET/OFI Only EFA provider is supported<br>
pytorch-1-8-gpu:844:844 [0] NCCL INFO NET/IB : No device found.<br>
pytorch-1-8-gpu:844:844 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1&lt;0&gt; [1]veth-app1-2:169.255.254.2&lt;0&gt;<br>
pytorch-1-8-gpu:844:844 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda11.1<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO Channel 00/02 :    0   1   2   3<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO Channel 01/02 :    0   1   2   3<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;3/-1/-1 [1] 3/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;3/-1/-1<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1 [1] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;2|2-&gt;3-&gt;-1/-1/-1 [1] -1/-1/-1-&gt;3-&gt;2|2-&gt;3-&gt;-1/-1/-1<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [1] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0)<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0)<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0)<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0)</p>
<p>pytorch-1-8-gpu:844:907 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device</p>
<p>pytorch-1-8-gpu:844:908 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO include/shm.h:41 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO include/shm.h:41 → 2</p>
<p>pytorch-1-8-gpu:844:908 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-68dbfe5912d345c9-0-0-1 (size 9637888)</p>
<p>pytorch-1-8-gpu:844:907 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-68dbfe5912d345c9-0-3-0 (size 9637888)</p>
<p>pytorch-1-8-gpu:844:910 [3] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO transport/shm.cc:101 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO transport/shm.cc:101 → 2</p>
<p>pytorch-1-8-gpu:844:909 [2] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO transport.cc:30 → 2<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO include/shm.h:41 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO transport.cc:30 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO include/shm.h:41 → 2<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO transport.cc:49 → 2</p>
<p>pytorch-1-8-gpu:844:910 [3] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-68dbfe5912d345c9-0-2-3 (size 9637888)</p>
<p>pytorch-1-8-gpu:844:907 [0] NCCL INFO transport.cc:49 → 2<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO transport/shm.cc:101 → 2<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO init.cc:766 → 2</p>
<p>pytorch-1-8-gpu:844:909 [2] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-68dbfe5912d345c9-0-1-2 (size 9637888)</p>
<p>pytorch-1-8-gpu:844:908 [1] NCCL INFO init.cc:840 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO init.cc:766 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO transport/shm.cc:101 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO init.cc:840 → 2<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO transport.cc:30 → 2<br>
pytorch-1-8-gpu:844:908 [1] NCCL INFO group.cc:73 → 2 [Async thread]<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO transport.cc:49 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO transport.cc:30 → 2<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO init.cc:766 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO transport.cc:49 → 2<br>
pytorch-1-8-gpu:844:907 [0] NCCL INFO group.cc:73 → 2 [Async thread]<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO init.cc:840 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO init.cc:766 → 2<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO init.cc:840 → 2<br>
pytorch-1-8-gpu:844:910 [3] NCCL INFO group.cc:73 → 2 [Async thread]<br>
pytorch-1-8-gpu:844:909 [2] NCCL INFO group.cc:73 → 2 [Async thread]<br>
pytorch-1-8-gpu:844:844 [0] NCCL INFO init.cc:906 → 2<br>
Traceback (most recent call last):<br>
File “test.py”, line 13, in <br>
output = model(data)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 918, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 166, in forward<br>
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 171, in replicate<br>
return replicate(module, device_ids, not torch.is_grad_enabled())<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 91, in replicate<br>
param_copies = _broadcast_coalesced_reshape(params, devices, detach)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 71, in _broadcast_coalesced_reshape<br>
tensor_copies = Broadcast.apply(devices, *tensors)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 23, in forward<br>
outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/comm.py”, line 58, in broadcast_coalesced<br>
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)<br>
RuntimeError: NCCL Error 2: unhandled system error</p>
</blockquote>

username: ptrblck
cooked: <p>Thanks for the update.<br>
The issue is caused by:</p>
<pre><code class="lang-python">NCCL WARN Call to posix_fallocate failed : No space left on device
</code></pre>
<p>which is raised e.g. if NCCL tries to allocate shared memory in <code>/dev/shm</code> and isn’t able to do so.<br>
I guess you might be using a container and are not properly configuring the shared memory usage (e.g. via <code>--ipc=host</code>).</p>