CUDA out of memory error when allocating one number to GPU memory

username: Asa-Nisi-Masa
cooked: <p>I am getting an out of memory error for CUDA when running the following code:</p>
<pre><code class="lang-auto">import torch

assert torch.cuda.is_available() == 1

x = torch.randn(1)

x.cuda()    # RuntimeError: CUDA error: out of memory
</code></pre>
<p>Running on GeForce GTX 750, Ubuntu 18.04. How can there not be enough memory for one float?</p>

username: ptrblck
cooked: <p>Could you check the current memory usage on the device via <code>nvidia-smi</code> and make sure that no other processes are running?<br>
Note that besides the tensor you would need to allocate the CUDA context on the device, which might take a few hundred MBs.</p>

username: Asa-Nisi-Masa
cooked: <p>Thank you for the response. Here is the <code>nvidia-smi</code> output. It seems that I only have ~300 MB left while idle. I probably do not have enough vRAM for the CUDA context that you mention.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b12078b27808e7d9868f1ce2e93d71e16e816082.png" data-download-href="https://discuss.pytorch.org/uploads/default/b12078b27808e7d9868f1ce2e93d71e16e816082" title="nvidia"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b12078b27808e7d9868f1ce2e93d71e16e816082_2_690x374.png" alt="nvidia" data-base62-sha1="pgW55vXGqtZ1R6FyOzRHa4n7y14" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b12078b27808e7d9868f1ce2e93d71e16e816082_2_690x374.png, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b12078b27808e7d9868f1ce2e93d71e16e816082.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b12078b27808e7d9868f1ce2e93d71e16e816082.png 2x" data-dominant-color="441A2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvidia</span><span class="informations">707×384 65.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Yeah, your GPU doesn’t seem to have enough memory, so you could try to run the code on e.g. Colab.</p>

username: Jeronimo_Molina
cooked: <p>Hello, … I’m getting same problem and I’m new on using local GPU.</p>
<p>I’ve using an RTX 3060, and when launching my code, two processes, python and a service I run are allocated un the VRAM (not only the models…):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757.png" data-download-href="https://discuss.pytorch.org/uploads/default/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757" title="Captura de pantalla de 2021-08-26 08-05-36"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757.png" alt="Captura de pantalla de 2021-08-26 08-05-36" data-base62-sha1="unvUqoSH0R2XFOtXLFloDW9QuPl" width="622" height="500" data-dominant-color="421B38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla de 2021-08-26 08-05-36</span><span class="informations">761×611 60.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you very much in advance</p>

username: ptrblck
cooked: <p>Are you seeing this memory usage by allocating a single scalar CUDATensor only?<br>
Based on the GPU memory usage for the Python process I would assume that you are running a larger workload.<br>
I don’t know what <code>...nesBasicas/env.bin/python</code> executes, but would generally recommend to close all unnecessary processes so that your main training script can use as much memory as possible.</p>

username: Jeronimo_Molina
cooked: <p>Thanks for your reply…</p>
<p>I’m  loading 4 (“only four…”) BERT models… yes the four models are really large… I’m working on Emotive Computing.</p>
<p>I’m not training ,… I’ve trained on collab and now I’m only loading the models for predicting (for now) on my local machine, …</p>
<p>the last process is a flask web service that loads the models and make predictions.</p>
<p>As I’m newby eith CUDA on a local machine,…</p>
<p>The first question is:</p>
<ul>
<li>I’m doing it correctly or I’m making some mistakes? Can I move python and the WS to the CPU memory (I think it is not possible…)?</li>
</ul>
<p>Thanks in advance</p>
<p>Jeronimo</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jeronimo_Molina" data-post="8" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeronimo_molina/48/41646_2.png" class="avatar"> Jeronimo_Molina:</div>
<blockquote>
<p>I’m loading 4 (“only four…”) BERT models… yes the four models are really large… I’m working on Emotive Computing.</p>
</blockquote>
</aside>
<p>Do you know how much memory a single model would take? Based on the description of your use case I guess you are working on an inference use case? My guess would be that 6GB are quite limited to host 4 BERT models, but it of course depends on the particular model architecture you are using.</p>
<aside class="quote no-group" data-username="Jeronimo_Molina" data-post="8" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeronimo_molina/48/41646_2.png" class="avatar"> Jeronimo_Molina:</div>
<blockquote>
<p>the last process is a flask web service that loads the models and make predictions.</p>
</blockquote>
</aside>
<p>Is this process loading other models as well, as it’s using GPU memory, too?</p>
<aside class="quote no-group" data-username="Jeronimo_Molina" data-post="8" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeronimo_molina/48/41646_2.png" class="avatar"> Jeronimo_Molina:</div>
<blockquote>
<p>Can I move python and the WS to the CPU memory (I think it is not possible…)?</p>
</blockquote>
</aside>
<p>All PyTorch objects are created on the CPU by default. Unless you push them manually to the GPU via <code>.cuda()</code> or <code>to('cuda')</code> the GPU won’t be used.</p>

username: Jeronimo_Molina
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Do you know how much memory a single model would take? Based on the description of your use case I guess you are working on an inference use case? My guess would be that 6GB are quite limited to host 4 BERT models, but it of course depends on the particular model architecture you are using.</p>
</blockquote>
</aside>
<p>I don’t know the memory usage of every model, but yes I need the 4 models to detect and infer emotions on texts (I use two “emotion classifier” models, one “sentimment analisys” model and one “toxocity classifier” model. May be 6 Gb is quite limmited.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Is this process loading other models as well, as it’s using GPU memory, too?</p>
</blockquote>
</aside>
<p>No this process is the only one that loads the models, make predictions using the models and returns the result using JSON</p>
<aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="74318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>All PyTorch objects are created on the CPU by default. Unless you push them manually to the GPU via <code>.cuda()</code> or <code>to('cuda')</code> the GPU won’t be used.</p>
</blockquote>
</aside>
<p>Exactly,… I load the models and I move to the GPU using “model.to(device)” where device is a var that if is there a GPU stores ‘cuda:0’ value and ‘cpu’ in other situation.<br>
But,… I assume that moving the models to the GPU the python ptrrocess is also moved to de GPU…</p>

username: ptrblck
cooked: <p>The Python process itself will not be moved to the GPU (GPUs cannot execute a Python engine) but it will initialize the CUDA context, load data (e.g. inputs, model parameters etc.) onto the GPU, and will launch CUDA kernels for the computation. <code>nvidia-smi</code> shows how much GPU memory the processes allocate on the device (the Python process itself is still being executed on the CPU).</p>

username: Jeronimo_Molina
cooked: <p>So, is correct that the two last processes of the image are allocated there?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757.png" data-download-href="https://discuss.pytorch.org/uploads/default/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757" title="Captura de pantalla de 2021-08-26 08-05-36"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e9b193be8657bfaf1ef6ee23c25cf0a4c70757.png" alt="Captura de pantalla de 2021-08-26 08-05-36" data-base62-sha1="unvUqoSH0R2XFOtXLFloDW9QuPl" width="622" height="500" data-dominant-color="421B38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla de 2021-08-26 08-05-36</span><span class="informations">761×611 60.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I don’t know what your Flask application is doing, but assuming it’s loading the model, pushes it to the GPU, calculates the predictions, and forward it to the user, I don’t know what the other “python” process is doing and if it’s created by the Flask app.</p>

username: Jeronimo_Molina
cooked: <p>My flask Application does exactly what you say.</p>
<p>The other python application  appears when I launch the flask app (so is mine), but  I d’ont know exactly wath is,… I think is the python interpreter. May be the python interpreter?</p>

username: ptrblck
cooked: <p>The other application is using the default <code>python</code> while the Flask app seems to use its own <code>python</code> shipped in an environment.<br>
In any case, I’m not familiar with Flask so don’t know if it’s supposed to spawn other processes and in fact why this process is using device memory.</p>

username: Jeronimo_Molina
cooked: <p>Ok, friend, thank you very very much for your help,…<br>
The two processes go to the GPU when I  translate the models to the GPU, so I thinbk it is necessary that they where on GPU, but I’ll research on it…</p>