Phantom PyTorch Data on GPU

actuallyaswin: <p>I have a Python script which is running a PyTorch model on a specific device, passed by using the <code>.to()</code> function and then the integer number of the device. I have four devices available so I alternate between 0, 1, 2, 3. This integer is set to a variable <code>args.device_id</code> that I use around my code.</p>
<p>I’m getting this situation where I see some phantom data appearing on a GPU device which I didn’t specify ever. See the picture below (this output is from <code>gpustat</code> which is a <code>nvidia-smi</code> wrapper):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36331062d378ce1740f03f6de79162a3cc3c70c0.png" alt="image" data-base62-sha1="7JtbCTg1yfZaU3ALxf8LNgx6xQ4" width="673" height="82"></p>
<p>Note that when I force-quit my script, all the GPUs return to zero usage. So there are no other programs running from other users.</p>
<p>I specifically added <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3">this snippet</a> to my main training/test loop, based on a previously-found PyTorch Discuss thread regarding finding-all-tensors. <code>gc</code> here is the Python garbage collection built-in module.</p>
<pre><code class="lang-auto">...
for obj in gc.get_objects():
    try:
        if (torch.is_tensor(obj) or
            (hasattr(obj, 'data') and torch.is_tensor(obj.data))):
            print(type(obj), obj.size(), obj.device)
    except:
        pass
...
</code></pre>
<p>Here is the result that I see on the command prompt:</p>
<pre><code class="lang-auto">2020-04-17 20:56:06 [INFO] Process ID: 3761, Using GPU device 1...
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 15872]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([51920]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([51920]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 51920]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 51920]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 51920]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 513, 203, 2]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 203, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 513, 203, 2]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 203, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 203, 513, 2]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 203, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1, 203, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([]) cpu
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([2048, 513]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([513, 512]) cuda:1
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 16000]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 16000]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 16000]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513, 2]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 63, 513]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000]) cuda:1
&lt;class 'torch.Tensor'&gt; torch.Size([1000, 2]) cuda:1
</code></pre>
<p>So everything on this list (except for one random outlier) all say “cuda:1”. How can I identify what data is sitting in “cuda:3” then? Appreciate the help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Another tid-bit, the phantom data always builds up to 577MiB exactly, never more than that. This behavior occurs even if I set the classic <code>os.environ['CUDA_VISIBLE_DEVICES']</code> variable at the top of the script.</p>

ptrblck: <p>Could you set the env variable in your terminal via:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=1,3 python script.py args
</code></pre>
<p>If you are using the <code>os.environ</code> method inside your script, you would have to make sure it’s called before any <code>torch</code> imports.</p>