CUDA error: peer mapping resources exhausted

jasperhyp: <p>Hi!  I’ve been trying to use <code>nn.DataParallel</code> for my model but am encountering this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 382, in &lt;module&gt;
    main(args, hparams)
  File "train.py", line 373, in main
    train(model, train_dataset, val_dataset, retrieved_tokens, args.output_dir, hparams, device)
  File "train.py", line 314, in train
    train_loss, train_accuracy, train_per_token_accuracy, train_perplexity = train_epoch(model, train_loader, retrieved_tokens, optimizer, hparams, device)
  File "train.py", line 236, in train_epoch
    result = model(x_tokens, retrieved, mask_tokens)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 158, in forward
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 175, in scatter
    return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 44, in scatter_kwargs
    inputs = scatter(inputs, target_gpus, dim) if inputs else []
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 36, in scatter
    res = scatter_map(inputs)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 23, in scatter_map
    return list(zip(*map(scatter_map, obj)))
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 19, in scatter_map
    return Scatter.apply(target_gpus, None, dim, obj)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/_functions.py", line 96, in forward
    outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams)
  File "/home/yeh803/.conda/envs/plm/lib/python3.8/site-packages/torch/nn/parallel/comm.py", line 189, in scatter
    return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))
RuntimeError: CUDA error: peer mapping resources exhausted
</code></pre>
<p>I tried using 8 GPUs and 4 GPUs and the problem persists. I also tried to <code>export NCCL_P2P_DISABLE=1</code>, but it didn’t work either.  I have 80GB free RAM so CPU mem shouldn’t be a problem. Any help would be much appreciated!</p>

ptrblck: <p>The error message indicates that you might be trying to enable more than 8 peer connections.<br>
Are you masking the 8 GPUs via <code>CUDA_VISIBLE_DEVICES</code>? If not, could any device try to enable more than 8 peer connections?</p>

jasperhyp: <p>There are 10 GPUs on my compute node and I’ve tried to set <code>os.environ['CUDA_VISIBLE_DEVICES']='0,1'</code> on the top of my python code to limit the GPUs that the model can see to less than 8. Should I do this in the system environment instead of in python code?</p>

jasperhyp: <p>Thank you, It seems to work when I prepend <code>CUDA_VISIBLE_DEVICES=0,1</code> before the script!  Interestingly, when I set it to <code>0,1,2,3,4,5,6,7,8,9</code>, it still worked, albeit only using GPUs 0~7.</p>
<p>Do you have any idea why I need to do this in the script instead of in the code though?  Also, would <code>DistributedDataParallel</code> also have such problem?  I am thinking of replacing DP with DDP.</p>

ptrblck: <aside class="quote no-group" data-username="jasperhyp" data-post="5" data-topic="167814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jasperhyp/48/58578_2.png" class="avatar"> jasperhyp:</div>
<blockquote>
<p>Do you have any idea why I need to do this in the script instead of in the code though?</p>
</blockquote>
</aside>
<p>You might be setting the env variable too late in your actual script. Once the CUDA context is created, this env variable won’t have any effect anymore, which is why I usually recommend to <code>export</code> it in your current terminal or prepend it to your <code>python</code> command in the terminal.</p>
<aside class="quote no-group" data-username="jasperhyp" data-post="5" data-topic="167814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jasperhyp/48/58578_2.png" class="avatar"> jasperhyp:</div>
<blockquote>
<p>I am thinking of replacing DP with DDP.</p>
</blockquote>
</aside>
<p>That is a good idea as <code>DP</code> suffers from some overheads in cloning the model’s <code>state_dict</code> in each forward pass as well as from an imbalanced GPU memory usage. <code>DDP</code> should thus give you a better performance.</p>

jasperhyp: <p>Thank you so much!  Very helpful.</p>