GPU is not utilized while occur RuntimeError: cuda runtime error: out of memory at

username: JisongXie
cooked: <p>I have encountered an odd problem:<br>
My lab has a server with four 1080Ti GPU(about 12G)，and it’s used by multiusers. I have installed CUDA9.0, cuDNN7.4.3 for CUDA9.0, pytorch 0.4.1。<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/af630bd3772cbc1e89378adc05df1cc594e2fbaf.png" alt="2" data-base62-sha1="p1xLiieSsUvvJ5azeekZV99opEH" width="362" height="80"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/bd8a39e682f2b7b43f561b5695d263326a9a2f21.png" alt="1" data-base62-sha1="r2KobDVWnPQ7k2xX0dm5i5UbycF" width="254" height="51"><br>
When I create a random tensor, neither small size of big size, when it’s print or copy, it occur an error, which say “cuda runtime error:out of memory”. I check the use of GPU, it’s not utilized and enough for such a tensor.<br>
create a tensor:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/808416aacd1aee8628c31a4e82224244ce6a3bb2.png" alt="3" data-base62-sha1="ikU76AuB05Ol6GiMq6K832wgYrU" width="564" height="142"><br>
print this tensor:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4c1420ad7bd2301ad0d5ba7af86cbbc7296f2b70.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c1420ad7bd2301ad0d5ba7af86cbbc7296f2b70" title="4"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4c1420ad7bd2301ad0d5ba7af86cbbc7296f2b70.png" alt="4" data-base62-sha1="aR1pu2aiZGgL0Ph57yjifkhzMXK" width="690" height="262" data-dominant-color="0B0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">891×339 11.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Then I think it may be the NVIDIA’s cuda problem, so I reinstall cuda of the same version, and cudnn. And then it works. I can successfully print it and copy it.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f2377cbfbeeb97803f803e92ef5ef24cf6c237bd.png" alt="7" data-base62-sha1="yyKqAt95JmxEJ9pzejUsu3Zj0Pb" width="620" height="431"><br>
Then I go to take a break, and when I go back to do the same thing, it fail. It occur the same problem as previous. The thing I need to add is, after I reinstall cuda and cudnn, my friend begin to run her tensorflow code with GPU0(I don’t know if it will affect).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cbd3afdf9afd5216892bedd1b88b4208f2a9f3ef.png" alt="5" data-base62-sha1="t58sKV5Xymt0hl5mAZalUqmTlMj" width="679" height="435"><br>
In addition, when I create the tensor and make it to GPU device, it really occupy the GPU usage, but when print or copy, it failed.For example, like the code I show above, I create a 10x10 size of tensor and make it to GPU2，then the GPU usage is as follow, it also seems wired, because it occupies too much.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/1/16b5947b6e87b9b9a7148ee87a3bd3f145ad1408.png" alt="6" data-base62-sha1="3eTxl0ChsfcEdjH8PYbyq8VSt4Y" width="677" height="455"></p>

username: albanD
cooked: <p>Hi,</p>
<p>Tensorflow has the bad habbit of taking <strong>all</strong> the memory on the device and prevent anything from happening on it as anything will OOM.<br>
There was a small bug in pytorch that was initializing the cuda runtime on device 0 when printing that has been fixed.<br>
A simple workaround is to use <code>CUDA_VISIBLE_DEVICES=2</code>. This will hide all devices but the one you specify and will make sure you never use other devices.</p>

username: JisongXie
cooked: <p>Awesome! Thank you, <a class="mention" href="/u/alband">@albanD</a>!<br>
In Python, I import os to set CUDA_VISIBLE_DEVICES, like this.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e2626006043a0c73f3912407a3969d70c7e54a60.png" alt="1" data-base62-sha1="wiGFUtPcYwNe54y8DIcSkLp34Mo" width="350" height="35"><br>
But when I set to use use GPU2 as previous, It doesn’t work.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ac7d9082e4beb4269b6d3a59c264c0bda929e5cb.png" alt="2" data-base62-sha1="oBV91JoZkXDzE8vXv4w2LbYr7uz" width="616" height="181"><br>
The error say “invalid device ordinal”. Then I realize that when I set the CUDA_VISIBLE_DEVICES, there are only the devices set is visible. For example, if I set GPU2 and GPU3 visible,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/71aad33118bb311e10f6300d164b702a5a309faf.png" alt="4" data-base62-sha1="gdxWArZdfvPvpSwMC86GBddL2zR" width="446" height="92"><br>
That means ‘cuda:0’ now refer to GPU2, while ‘cuda:1’ refer to GPU3. Therefore ‘cuda:2’ will make error.<br>
Thanks again to <a class="mention" href="/u/alband">@albanD</a>! You help me a lot!<br>
I would have replied to you soon, but these few day I can’t access to the forum.</p>

username: JessicaDuFirst
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ad30b14a6cc0adea4187ad8983651a19c7ec440f.png" alt="Torch1" data-base62-sha1="oI6VoiSigIeo9Zov6LmSrBLHufZ" width="580" height="141"> ![torch2|690x86]</p>
<p>I meet the the problem, but the answer above couldn’t solve my problem, who can help me ?</p>

username: JessicaDuFirst
cooked: <p>i meet the same problem , but your answer couldn’t work.<br>
GPU全部空闲，但还是报错内存不足</p>

username: JisongXie
cooked: <p>Please provide more information, otherwise it’s difficult to know where is your bug. How many gpus you have, and how you write the code to copy to cuda, and maybe your batch size is too large?</p>
<p>By the way, it would be better not to use Chinese to post your problem. It’s not beneficial for you to seek a solution. Therefore, please use English, to get it caught by more viewers.</p>

username: JessicaDuFirst
cooked: <p>Thanks a lot, i changed the bathsize smaller, then fix the problem</p>

username: cycyco
cooked: <p>I have the same problem too. However, there is nobody using tensorflow at the same time. It is wired.</p>