CUDA error: out of memory when there is enough memory?

111167: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a24ed0986b5d8f1b6778097635c2087c95502e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8a24ed0986b5d8f1b6778097635c2087c95502e" title="截屏2021-07-30 20.25.46"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a24ed0986b5d8f1b6778097635c2087c95502e.png" alt="截屏2021-07-30 20.25.46" data-base62-sha1="o3NXoIIrRGMJq3JdTJOYjfFBLBQ" width="690" height="93" data-dominant-color="0C323C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-07-30 20.25.46</span><span class="informations">1108×150 8.28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/5597b7d674c0509e6a9fc74b35b77b0898a0bfcf.png" data-download-href="https://discuss.pytorch.org/uploads/default/5597b7d674c0509e6a9fc74b35b77b0898a0bfcf" title="截屏2021-07-30 20.25.37"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/5597b7d674c0509e6a9fc74b35b77b0898a0bfcf.png" alt="截屏2021-07-30 20.25.37" data-base62-sha1="cdbDErkpOFP5qMkVmWHD7BI2ltB" width="690" height="230" data-dominant-color="0C313B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-07-30 20.25.37</span><span class="informations">1032×344 23.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to solve the above problems? Thanks!!</p>

ptrblck: <p>The error shouldn’t be raised, if your device has enough memory.<br>
Could you explain a bit more why you think there is enough memory and how you’ve estimated the memory requirement needed in your script?</p>

111167: <p>I confirmed that gpu-8 is not used through the nvidia-smi command.</p>
<p>I just created a simple torch.Tensor([1.0]).cuda() will display “RuntimeError: CUDA error: out of memory”.</p>

111167: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I use the nvidia-smi, the output is as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de70227e00d2151620aebd5f665941667535c263.png" data-download-href="https://discuss.pytorch.org/uploads/default/de70227e00d2151620aebd5f665941667535c263" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de70227e00d2151620aebd5f665941667535c263.png" alt="image" data-base62-sha1="vJMffD6y1LeEnyEHRYGXs82aSdl" width="690" height="458" data-dominant-color="30323D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">728×484 9.67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then, I tried to creat a tensor on gpu with<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1afe439406052b6f4936c44c097308212a27017c.png" alt="image" data-base62-sha1="3QNaGhJUejZ9f7wUJve4JtMzKpK" width="431" height="300"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46115de149ad7f1b4ea0e577ec1b241f41d0f0f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/46115de149ad7f1b4ea0e577ec1b241f41d0f0f4" title="tsts"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46115de149ad7f1b4ea0e577ec1b241f41d0f0f4.png" alt="tsts" data-base62-sha1="9ZQD7ofdnjogLIXCGwxXDi4zbzC" width="644" height="499" data-dominant-color="68565F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tsts</span><span class="informations">727×564 37.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It can be seen that gpu-0 to gpu-7 can successfully apply for tensor, but gpu-8 and gpu-9 will have a “cuda out of memory” error, even if there is sufficient memory.</p>