RuntimeError: Tensors must be CUDA and dense

username: Huige_Cheng
cooked: <p>I face this when using torch.nn.parallel.DistributedDataParallel(pytorch 1.4.0), and also using below</p>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
tensor = torch.zeros(*shape, device=device).scatter_add(1, segment_ids, data)</p>
<p>File “/home/gezi/mine/pikachu/utils/melt/eager/train.py”, line 1398, in train<br>
loss.backward()<br>
File “/home/gezi/env/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/gezi/env/anaconda3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Tensors must be CUDA and dense</p>
<p>How to slove this ? I tried many such as<br>
tensor = torch.zeros(*shape).cuda().scatter_add(1, segment_ids, data)<br>
but this only works for DataParallel not DistributedDataParallel.</p>
<p>Another problem of DIstributedDataParallel is each process using all gpus like below, is this by design ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c64603e6055762df113d433817ea8cbc4b0ad5e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/c64603e6055762df113d433817ea8cbc4b0ad5e5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c64603e6055762df113d433817ea8cbc4b0ad5e5_2_352x500.png" alt="image" data-base62-sha1="si0xR9bIrnFcuPsWOBG66pJb5wV" width="352" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c64603e6055762df113d433817ea8cbc4b0ad5e5_2_352x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c64603e6055762df113d433817ea8cbc4b0ad5e5_2_528x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c64603e6055762df113d433817ea8cbc4b0ad5e5_2_704x1000.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1818×2576 467 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Huige_Cheng" data-post="1" data-topic="72239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huige_cheng/48/13258_2.png" class="avatar"> Huige_Cheng:</div>
<blockquote>
<p>RuntimeError: Tensors must be CUDA and dense</p>
</blockquote>
</aside>
<p>Are you using sparse tensors?</p>
<blockquote>
<p>Another problem of DIstributedDataParallel is each process using all gpus like below, is this by design ?</p>
</blockquote>
<p>How did you construct DDP? You need to either set <a href="https://github.com/pytorch/pytorch/blob/v1.4.0/torch/nn/parallel/distributed.py#L171" rel="noopener nofollow ugc"><code>device_ids</code></a> arg properly or use the <code>CUDA_VISIBLE_DEVICES</code> env var to configure that, and make sure no DDP processes share the same GPU. Otherwise, each process will try to use all visible devices, and when two DDP process share a GPU, it could hang.</p>

username: Huige_Cheng
cooked: <ol>
<li>Yes I’m using Embedding with arg sparse=True. But seems not ok to run DDP only if I using scatter_add later.</li>
<li>If using 2 processes to run DDP. Then I set CUDA_VISIBLE_DEVICE=0,1 for each prcoess.<br>
code like below<br>
rank = dist.get_rank()<br>
device = torch.device(‘cuda’, rank)<br>
model = model.to(device)<br>
model = torch.nn.parallel.DistributedDataParallel(model,device_ids=[rank],output_device=rank)<br>
I tried to launch each process with CUDA_VISIBLE_DEVICE=0 CUDA_VISIBLE_DEVICE=1 but seems not work.</li>
</ol>

username: Huige_Cheng
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Well the second problem is due to I’m using tf dataset eager mode to read data first then convert to torch tensors, the problem has been solved.<br>
For the first I find yes it is due to using sparse not related to scatter_add. So the problem is the same as <a href="https://discuss.pytorch.org/t/distributeddataparallel-sparse-embeddings/60410" class="inline-onebox">DistributedDataParallel Sparse Embeddings</a></p>