Bug in DataParallel? Only works if the dataset device is cuda:0

rasbt: <p>Hi all,</p>
<p>when I am using <code>DataParallel</code>, I notice that it only works when the data is on the first available cuda device.</p>
<p>Say I am doing something like</p>
<pre><code class="lang-python">    for batch_idx, (features, targets) in enumerate(train_loader):
        
        features = features.to(device)
        targets = targets.to(device)
</code></pre>
<p>If device is “cuda:1” it throws an error <code>RuntimeError: all tensors must be on devices[0]</code>. However, if I change the device to “cuda:0” the parallel training on multiple GPUs works just fine.</p>
<p>Is that on purpose or a bug? If this is a current limitation, I feel like it should be documented at <a href="https://pytorch.org/docs/stable/nn.html?highlight=dataparallel#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=dataparallel#torch.nn.parallel.DistributedDataParallel</a></p>
<p>Happy to send a PR, just wanted to know if I am understanding this correctly …</p>

cyberwillis: <p>Hi</p>
<p>How are you setting your cuda device ? What Gpus are you using ?</p>
<p>In general to distribute equally to all your GPUs you need to pass “cuda”. By passing “cuda:0” it will only use the GPU 0 not all of them.</p>
<p>On the past I had a GTX 690 (DUAL CORE GPU already in SLI) that only was controlled by the GPU:0 even NVIDIA developers told me that was a bug on the driver. But never fixed.</p>

ptrblck: <p>Could you post the code part where your model is wrapped into <code>DataParallel</code>?<br>
As far as I know, you have to push the data onto the “default” device, i.e. the GPU where the scatter and gather will take place. If there is a mismatch, this error is thrown.</p>

rasbt: <blockquote>
<p>In general to distribute equally to all your GPUs you need to pass “cuda”. By passing “cuda:0” it will only use the GPU 0 not all of them.</p>
</blockquote>
<p>of course. this was just for the dataset. Let be paste a bit more code for the context</p>
<blockquote>
<p>Could you post the code part where your model is wrapped into  <code>DataParallel</code> ?</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">
device = torch.device("cuda:0")

...

model = VGG16(num_features=num_features, num_classes=num_classes)
if torch.cuda.device_count() &gt; 1:
    print("Using", torch.cuda.device_count(), "GPUs")
    model = nn.DataParallel(model)

model.to(device)

...

for epoch in range(num_epochs):
    
    model.train()
    for batch_idx, (features, targets) in enumerate(train_loader):
        
        features = features.to(device)
        targets = targets.to(device)

</code></pre>
<p>In this case, it’s 4 GPUs, and the whole things seems to work; I get even 3x speed-up compared to the 1 GPU version without DataParallel. It’s also utilizing the default GPU 99% and the others ~95%. It only works if <code>"cuda:0"</code> for the device, not <code>"cuda:1"</code>  (with cuda devices 0, 1, 2, 3 available; if it’s weird why I am asking for that: I have left the monitor plugged in into the cuda:0 device, and since it’s weekend, I am working from home :P).</p>

ptrblck: <p>Yeah, I had similar “use cases” in the past. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Could you try to set <code>output_device=device</code> in <code>nn.DataParallel</code> and set <code>device='cuda:1'</code>?</p>

rasbt: <p>Thanks, I tried that, still the same issue. In fact, I tried all different combinations</p>
<p><code>device=torch.device('cuda:1')</code>, <code>device='cuda:1'</code> (when using <code>output_device=device</code>, and transferring models and features)</p>
<p>as well</p>
<p>as <code>output_device=1</code> Still the same issue.</p>

rasbt: <blockquote>
<p>By passing “cuda:0” it will only use the GPU 0 not all of them.</p>
</blockquote>
<p>I think if <code>"cuda:0"</code> is your default/current device, the shouldn’t be a difference between <code>torch.device("cuda")</code> and <code>torch.device("cuda:0")</code> according to the docs at <a href="https://pytorch.org/docs/stable/tensor_attributes.html?highlight=torch%20device#torch-device" class="inline-onebox" rel="noopener nofollow ugc">Tensor Attributes — PyTorch 2.1 documentation</a></p>
<p>The parallelism (using all GPUs) is basically achieved via <code>model = nn.DataParallel(model)</code> where it uses all GPUs by default, except you specify only a subset via the <code>device_ids</code> argument of DataParallel).</p>
<blockquote>
<p>What Gpus are you using ?</p>
</blockquote>
<p>that would be 4 GeForce 1080Tis. They are all the same, but maybe there’s sth weird about the first device regarding the drivers like you said …</p>

cyberwillis: <p>Hi again…</p>
<p>What I tried to say was …</p>
<pre><code class="lang-auto">device = ("cuda" if torch.cuda.is_available() else "cpu" )
model = net()
if torch.cuda.device_count() &gt; 1:
    # device_ids has a default : all
    model = torch.nn.DataParallel(model, device_ids=[0, 1, 2, 3]) 
model.to(device)
</code></pre>

rasbt: <aside class="quote no-group" data-username="cyberwillis" data-post="2" data-topic="28634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cyberwillis/48/6872_2.png" class="avatar"> cyberwillis:</div>
<blockquote>
<p>Gpus are you using ?</p>
<p>In general to distribute equally to all your GPUs you need to pass “cuda”. By passing “cuda:0” it will only use the GPU 0 not all of them.</p>
</blockquote>
</aside>
<p>Thanks, I tried that, but then it is still using the cuda:0 device for the data gathering, because on my setup, your example would be equivalent to</p>
<pre data-code-wrap="python"><code class="lang-python">device = ("cuda:0" if torch.cuda.is_available() else "cpu" )
model = net()
if torch.cuda.device_count() &gt; 1:
    # device_ids has a default : all
    model = torch.nn.DataParallel(model, device_ids=[0, 1, 2, 3]) 
model.to(device)
</code></pre>
<p>(PS: It’s correctly utilizing all 4 GPUs based on what I can see via nvidia-smi, but the gradient averaging etc. is carried out on the first card)</p>
<p>however, I want to use <code>"cuda:1"</code> (or 2, or 3) for the data gathering instead of the 1st device. It’s not super important to me, since I can simply plug in the monitor into a different card, but the discussion is also to see whether there’s a bug in PyTorch.</p>

cyberwillis: <p>Then you can try this…<br>
Letting device = “cuda” is generic in case of the pure model setting the first one GPU or in case of DataParallel you need send the exact ids you want or just omit it to get them all</p>
<pre><code class="lang-auto">device = ("cuda" if torch.cuda.is_available() else "cpu" )
model = net()
if torch.cuda.device_count() &gt; 1:
    # device_ids has a default : all
    model = torch.nn.DataParallel(model, device_ids=[1,2]) 
model.to(device)
</code></pre>

cyberwillis: <p>I do have another question for you.</p>
<p>What version are you using? I am using the <strong>recent</strong> commit of PyTorch what about you ?</p>

rasbt: <p>Hm, but I still want to use all available GPUs. E.g., to better illustrate what I mean, let me use this DataParallel illustration from <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255:" class="inline-onebox" rel="noopener nofollow ugc">Medium</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/623cbf5248b1ccc24d4d4895bfefee98b8abd682.png" data-download-href="https://discuss.pytorch.org/uploads/default/623cbf5248b1ccc24d4d4895bfefee98b8abd682" title="22%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/623cbf5248b1ccc24d4d4895bfefee98b8abd682_2_690x244.png" alt="22%20PM" data-base62-sha1="e12WrJIEQWs26ITkEDlXsSmAOEa" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/623cbf5248b1ccc24d4d4895bfefee98b8abd682_2_690x244.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/623cbf5248b1ccc24d4d4895bfefee98b8abd682_2_1035x366.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/623cbf5248b1ccc24d4d4895bfefee98b8abd682.png 2x" data-dominant-color="DFE4E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">22%20PM</span><span class="informations">1113×395 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Basically, in the ones that I circled, I want this to be e.g., GPU-2 not GPU-1. Theoretically, this should be possible by just setting  <code>device=torch.device('cuda:1')</code></p>
<p>and then using</p>
<ol>
<li><code>.to(device)</code> for the model and the data tensors.</li>
<li>As well as the <code>output_device=device</code> inside <code>DataParallel</code></li>
</ol>
<p>PS: I am using PyTorch 0.4.1, that’s the most recent one I think (except for the release candidates)</p>

cyberwillis: <p>Now I get it</p>
<p>Yeah I know this article from this blog.</p>
<p>I believe the problem may be a bug on your version of pytorch that is too old.<br>
I am using 1.0.0a0+479517<br>
Just to mention before 1.0 was a 0.5 preview…</p>

rasbt: <p>Good that you bring up the version issue. I will upgrade at a later point once the stable release comes out and then revisit that topic.</p>

cyberwillis: <p>If you have time you can build it or try to use the nightly build to try it. I believe is on the official web site. If not you still can build it. It will take a few hours depending of memory and cores on CPU … but you can try.</p>

rasbt: <p>Yeah. But I probably will wait a bit since I currently have a well working custom compiled version of the stable one. It’s kind of a bit of work for me so I will wait until the next release comes out. This feature is not so urgent for me personally, but I suspected that there might be a bug in the PyTorch implementation when using DataParallel, because according to the docs, changing the default device (like the ones circled in the figure) should be possible.</p>

InnovArul: <p>What if you say <code>device_ids=[1, 2, 3,0]</code> in nn.DataParallel? Or <code>export CUDA_VISIBLE_DEVICES='1,2,3,0'</code> in command prompt?</p>

rasbt: <blockquote>
<p>What if you say  <code>device_ids=[1, 2, 3,0]</code>  in nn.DataParallel?</p>
</blockquote>
<p>Ha! This solves the issue! So when I define</p>
<pre><code>device = torch.device("cuda:1")
</code></pre>
<p>for</p>
<pre><code>model.to(device)
</code></pre>
<p>(and the data tensors) it breaks when I leave the defaults in <code>DataParallel</code> or have sth like <code>device_ids=[0, 1, 2, 3]</code> in <code>DataParallel</code> but if I then change it  to  <code>device_ids=[1, 2, 3, 0]</code> it works. Just wondering if this is intended or a bug. <a class="mention" href="/u/smth">@smth</a></p>

InnovArul: <p>I will let the expert (<a class="mention" href="/u/smth">@smth</a>) to answer you.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"> IMHO it is intended to be this way.</p>

rasbt: <p>makes sense then though. Just feel like it should be in the docs then. Happy to send a PR with an “explanation” if that’s indeed just a misunderstanding <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>