PyTorch OOM with one-element CUDA Tensor while a GPU card is free after killing all python processes

username: jianchao-li
cooked: <p>Recently I ran into a weird problem when using PyTorch. I have 8 GPU cards in the machine. After running a PyTorch training program for some time, I stopped it by Ctrl+C and then I checked the cards using <code>nvidia-smi</code>. Everything looked good.</p>
<pre data-code-wrap="bash"><code class="lang-bash">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.46                 Driver Version: 390.46                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:1A:00.0 Off |                    0 |
| N/A   32C    P0    25W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00000000:1F:00.0 Off |                    0 |
| N/A   34C    P0    25W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-PCIE...  Off  | 00000000:20:00.0 Off |                    0 |
| N/A   33C    P0    25W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-PCIE...  Off  | 00000000:21:00.0 Off |                    0 |
| N/A   33C    P0    23W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-PCIE...  Off  | 00000000:B2:00.0 Off |                    0 |
| N/A   32C    P0    26W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-PCIE...  Off  | 00000000:B3:00.0 Off |                    0 |
| N/A   35C    P0    26W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-PCIE...  Off  | 00000000:B4:00.0 Off |                    0 |
| N/A   34C    P0    25W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-PCIE...  Off  | 00000000:B5:00.0 Off |                    0 |
| N/A   35C    P0    25W / 250W |     11MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>I also wrote a <code>check.cu</code> to check the GPU memory.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include &lt;iostream&gt;
#include "cuda.h"
#include "cuda_runtime_api.h"
  
using namespace std;
  
int main( void ) {
    int num_gpus;
    size_t free, total;
    cudaGetDeviceCount( &amp;num_gpus );
    for ( int gpu_id = 0; gpu_id &lt; num_gpus; gpu_id++ ) {
        cudaSetDevice( gpu_id );
        int id;
        cudaGetDevice( &amp;id );
        cudaMemGetInfo( &amp;free, &amp;total );
        cout &lt;&lt; "GPU " &lt;&lt; id &lt;&lt; " memory: free=" &lt;&lt; free &lt;&lt; ", total=" &lt;&lt; total &lt;&lt; endl;
    }
    return 0;
}
</code></pre>
<p>The output also looked good.</p>
<pre data-code-wrap="bash"><code class="lang-bash">GPU 0 memory: free=16488464384, total=16945512448
GPU 1 memory: free=16488464384, total=16945512448
GPU 2 memory: free=16488464384, total=16945512448
GPU 3 memory: free=16488464384, total=16945512448
GPU 4 memory: free=16488464384, total=16945512448
GPU 5 memory: free=16488464384, total=16945512448
GPU 6 memory: free=16488464384, total=16945512448
GPU 7 memory: free=16488464384, total=16945512448
</code></pre>
<p>Then I moved forward to try to create a one-element CUDA Tensor. And an OOM error happened.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import numpy as np
 
if __name__ == '__main__':
    x = np.random.randn(1)
    try:
        t = torch.cuda.FloatTensor(x)
        print('Success!')
    except Exception as e:
        print(e)
</code></pre>
<p>GPU 2 seemed to be OOM.</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ CUDA_VISIBLE_DEVICES=0 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=1 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=2 python3 check.py 
CUDA error: out of memory
$ CUDA_VISIBLE_DEVICES=3 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=4 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=5 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=6 python3 check.py 
Success!
$ CUDA_VISIBLE_DEVICES=7 python3 check.py 
Success!
</code></pre>
<p>I followed <a class="mention" href="/u/smth">@smth</a>’s suggestion in <a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775/14">this reply</a> and killed all the python processes.</p>
<aside class="quote no-group" data-username="smth" data-post="14" data-topic="13775" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775/14">PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error</a></div>
<blockquote>
<p><span class="mention">@rabst</span><br>
so, I remember this issue. When investigating, we found that there’s actually a bug in python multiprocessing that might keep the child process hanging around, as zombie processes.<br>
It is not even visible to <code>nvidia-smi</code> .</p>
<p>The solution is <code>killall python</code> , or to <code>ps -elf | grep python</code> and find them and <code>kill -9 [pid]</code> to them.</p>
</blockquote>
</aside>
<p>But the above problem still occurred. I tried to reinstall PyTorch and it did not fix the problem. I once tried to restart the machine and it worked. But it would not be feasible to restart the machine every time since I was running it on a server…</p>
<p>Could anyone please shed some lights on this problem? By the way, I was using PyTorch 0.4.1 and CUDA 9.0. And the program I ran was <a href="https://github.com/CSAILVision/semantic-segmentation-pytorch" rel="noopener nofollow ugc">semantic-segmentation-pytorch</a>.</p>

username: SimonW
cooked: <p>There could have been a bug on uppercase <code>Tensor</code> ctor in 0.4.1 that is now fixed on master. Any case, you shouldn’t use the upper case ctors. Could you try <code>torch.randn(1, device='cuda')</code> instead and see if you still OOM?</p>
<p>It’s not related to zombie processes. Otherwise it will show up in nvidia-smi as occupied memory.</p>

username: jianchao-li
cooked: <p>Hello, <a class="mention" href="/u/simonw">@SimonW</a>. Thank you for your reply.</p>
<p>So I changed the codes to be as follows.</p>
<pre><code class="lang-python">import torch
 
if __name__ == '__main__':
    try:
        t = torch.randn(1, device='cuda')
        print('Success!')
    except Exception as e:
        print(e)
</code></pre>
<p>It still said OOM on GPU 2.</p>
<pre><code class="lang-bash">$ CUDA_VISIBLE_DEVICES=2 python3 check.py 
CUDA error: out of memory
</code></pre>

username: SimonW
cooked: <p>Looks like there is something wrong with your CUDA? I don’t know. It’s unlikely a pytorch problem.</p>

username: jianchao-li
cooked: <p>Hi. Thank you for your reply. I also tried to initialize some data on that GPU card using Tensorflow and it also complained about OOM.</p>
<p><a class="mention" href="/u/ngimel">@ngimel</a> Could you please shed some lights on this issue? Thank you.</p>

username: Preetham_Gali
cooked: <p>I am so facing same issue, <a class="mention" href="/u/jianchao-li">@jianchao-li</a> did you find any solution other than restarting?</p>