Sending tensor to cuda commits lots of memory that is never released

Chris_Oosthuizen: <p>On empty jupyter notebook</p>
<p>Current memory use 2.6GiB of 7.7 GiB as seen on Ubuntu System Monitor</p>
<blockquote>
<p>a = torch.tensor(1).cuda(0)</p>
</blockquote>
<p>After running the command above memory use jumps to 4.7GiB<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc632d12045838ffffa70f9d47a0298035a219c9.png" alt="image" data-base62-sha1="qSyfnWCp4owlqKsTMgX1NrGQLfb" width="414" height="138"><br>
I’m not able to release this memory other than restarting the kernel. Tried <code>del a</code>, working on cuda in a function and torch.cuda.empty_cuda() but these do not work.</p>
<p>This only happens the first time I send tensor to cuda. To me this use of memory is ironic as I’m working on GPU and it limits how much I can send to cuda as the RAM committed does grow when batch sizes grow.</p>
<p>Working with tensor on .cpu() has none of these memory commitment issues.</p>
<p>My questions are:</p>
<ol>
<li>Why is so much memory being committed when sending tensor with single INT tensor to GPU?</li>
<li>How can I release this memory again without killing the kernel?</li>
</ol>

ptrblck: <p>Sending a tensor to the GPU should not allocate that much system RAM.<br>
Could you post some information about your setup?<br>
I.e. which GPU are you using as well as PyTorch version, how you’ve installed it (built from source or binaries), as well as the local CUDA and cudnn versions, if installed.</p>
<p>I suspect some just-in-time compilation might be going on in the background.<br>
How long does this command take when you first run it?</p>

Chris_Oosthuizen: <p>It takes a second or so to run the first time.</p>
<p>Linux-x86_64<br>
NVidia Driver 430.64<br>
GeForce RTX 2070<br>
torchvision               0.3.0           py37_cu10.0.130_1    pytorch installed with conda as part of fastai install.<br>
Cuda compilation tools, release 10.1, V10.1.105<br>
Having trouble verifying version of CudNN</p>

ptrblck: <p>Could you post the PyTorch version via <code>print(torch.__version__)</code>?</p>

Chris_Oosthuizen: <p>Thank you for your time on this</p>
<aside class="quote no-group quote-modified" data-username="ptrblck" data-post="4" data-topic="69181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(torch.<strong>version</strong>)</p>
</blockquote>
</aside>
<p>1.1.0</p>

ptrblck: <p>Could you update to the latest stable release and rerun the code, please?</p>

Chris_Oosthuizen: <p>I was able to get 1.3.1 on jupyter notebook and 1.4.1 to work with python3 in terminal with different conda environments.<br>
The memory still goes up from 2.7 to 4.7GiB after running the code</p>

ptrblck: <p>Could you run <code>perf top</code> while executing this line of code and check for any ptx calls?</p>

Chris_Oosthuizen: <p>I’ve managed to install perf top, but am not sure how to check for ptx calls.<br>
On running it I can see libcuda.so.430.64  dominating with 15% overhead when I run the .cuda(0) command for the first time</p>

Chris_Oosthuizen: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> does the above answer your question? Sorry to bump. Still haven’t been able to resolve my issue.</p>