Out of memory when running model.cuda()

fanl: <p>I got a strange problem when training models:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "tools/train.py", line 243, in &lt;module&gt;
    main()
  File "tools/train.py", line 227, in main
    meta=meta)
  File "mmdet3d/apis/train.py", line 34, in train_model
    meta=meta)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/mmdet/apis/train.py", line 79, in train_detector
    model.cuda(),
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 637, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 552, in _apply
    param_applied = fn(param)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 637, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
</code></pre>
<p>The OOM error is raised when moving the model to GPU (model.cuda()). There is no GPU resources competition.<br>
Since I use the standard training config, it is impossible to produce such a model large enough to consume ~11 GB of memory.<br>
The issue seems related to the running environment, because it disappears after switching to another server (with the same image).<br>
Though it is solved, I have no idea for the real reasons. I can’t understand it at all.<br>
Does anyone know the reasons?</p>

ptrblck: <p>Based on your description the error message sounds like a red herring and you might indeed be hitting a setup issue. I don’t know what differences are between the environments, so can’t comment on a potential root cause.<br>
In the past I’ve seen “random” CUDA errors if you were already hitting a sticky CUDA error, the CUDA context would thus be invalid, and would then try to run more CUDA operations.</p>

fanl: <p>Hi<br>
I think you are right, and it is indeed a red herring. The root cause might be related to hardware. The following code snippet can reproduce the bug:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.rand(1).cuda(0)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
&gt;&gt;&gt; torch.rand(1).cuda(1)
tensor([0.7275], device='cuda:1')
</code></pre>
<p>It seems that there are some problems with GPU 0. Do you have any idea?</p>

ptrblck: <p>Try to reboot the machine first (or reset the device) and see, if this could help.<br>
Hardware issues are certainly possible, but the majority of issues is usually created on the software side.</p>