Strange behavior nn.Dataparallel

111233: <p>Hi all! I have 4 GPUs 1080 Ti, and when I run training inception_v3 net on multiple GPU model have strange behavior. I didnt rewrite my code much from training on 1 GPU, just add:<br>
model = nn.DataParallel(model, device_ids=[0,1,2,3]).cuda()<br>
When I run script with device_ids=[0,1] GPUs full utilized and train much faster, when I run script with device_ids=[0,1,2] or device_ids=[0,1,2, 3] script starting (GPU full utilized in nvidia-smi, but reserved memory on card small: 1Gb on first card, and 500 Mb on other) and model dont train. Where I am wrong?  And I have 2 cores processor. Thanks for response. Sorry for my English</p>

ptrblck: <p>Does your code just hangs using all 4 GPUs or does “model doesn’t train” mean that the training loss is worse than on a single device?</p>

111233: <p>With 1 or 2 GPUs model train, loss decreased, all good. With 3 or 4 GPUs sript run, but train dont work, I logged every epoch loss and accuracy. I run sript overnight, but nothing logged.</p>

ptrblck: <p>Could you try to run the code only on device 2 and 3, if 0 and 1 are working?<br>
Set the device via <code>.to('cuda:2')</code> or <code>.to('cuda:3')</code>.</p>

111233: <p>If I set torch.device(“cuda:2”) or torch.device(“cuda:3”), I got error: tensors must be on the same device. If I set nn.DataParallel(model, device_ids=[1,2,3]).cuda(), on first GPU (with index 0) free memory decreased (same as I run training on it) and after that raise error: tensors must be on the same device. In training block of code batch of images send to GPU (input.to(device)). May be this happen because processor have only 2 cores?</p>

ptrblck: <p>Could you use device ids 0 and 1 in your script for <code>nn.DataParallel</code> and launch the script via:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=2,3 python script.py args
</code></pre>

111233: <p>Running script with this parameters launch script, but used GPU memory on device 2 and 3: 1Gb and 500 Mb and model dont training.</p>

ptrblck: <p>Thanks for the test.<br>
Could you run the code on a single device now and check, if it’s working on GPU2 and GPU3?</p>

111233: <p>Yes, if set in terminal CUDA_VISIBLE_DEVICE=2, or 3 training is running ok.</p>

ptrblck: <p>So you only see the hand when you are using <code>nn.DataParallel</code> with device 2 and 3?<br>
Could you run the <code>p2pBandwidthLatencyTest</code> from the CUDA samples?</p>

111233: <p>p2pBandwidthLatencyTest works 24 hours and dont done.<br>
P2P Connectivity Matrix<br>
D\D     0     1     2     3<br>
0       1     1     1     1<br>
1       1     1     1     1<br>
2       1     1     1     1<br>
3       1     1     1     1<br>
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)<br>
D\D     0      1      2      3<br>
0 352.55   0.41   0.41   0.41<br>
1   0.39 216.53   0.39   0.39<br>
2   0.39   0.39 349.40   0.39<br>
3   0.39   0.39   0.39 350.65<br>
Unidirectional P2P=Enabled Bandwidth Matrix (GB/s)<br>
D\D     0      1      2      3<br>
0 353.51   0.41   0.41   0.41<br>
1   0.35 376.32   0.00   0.00<br>
2   0.21   0.00 376.32   0.00<br>
3   0.21   0.00   0.00 377.78<br>
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)<br>
D\D     0      1      2      3<br>
0 374.52   0.68   0.68   0.69<br>
1   0.69 375.24   0.67   0.67<br>
2   0.69   0.68 374.52   0.68<br>
3   0.69   0.68   0.68 374.70<br>
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)<br>
D\D     0      1      2      3<br>
0 377.23   0.68   0.68   0.68<br>
1   0.68 372.56   0.55   0.55<br>
2   0.67   0.55 370.96   0.55<br>
3   0.67   0.55   0.55 370.26<br>
P2P=Disabled Latency Matrix (us)</p>

ptrblck: <p>Thanks for the information. This points towards some communication issues between the GPUs.<br>
Could you run the PyTorch code using <code>NCCL_P2P_DISABLE=1</code> to use shared memory instead of p2p access?</p>

111233: <p>Thanks! this option works for me.</p>