How to calculate the GPU memory that a model uses?

may1: <p>I wanted to reduce the size of Pytorch models since it consumes a lot of GPU memory and I am not gonna train them again.</p>
<p>First, I thought I could change them to TensorRT engine.<br>
and then I was curious how I can calculate the size of gpu memory that it uses.</p>
<p>Pytorch model size can be calculated by</p>
<pre><code class="lang-auto">torch.cuda.memory_allocated
</code></pre>
<p>or<br>
calculating using model.parameters() and model.buffers()</p>
<p>I checked if the above results had same values and they had.</p>
<p>But the size of TensorRT engine or other Scripted modules for GPU cannot be calculated by the above torch functions.<br>
So I thought I could check the gpu memory usage size with GPUtil library.</p>
<p>However, the memory usage size that was calculated by GPUtil library (using nvidia-smi) was too different.<br>
For example, one model has 13MiB size but almost 2 GiB was allocated in GPU. The other model has 171MiB but also around 2GiB was allocated in GPU. I didn’t put other objects such as inputs in GPU.</p>
<p>and Even after deleting the model,</p>
<pre><code class="lang-auto">del model
gpu = GPUtil.getGPUs()[0]
memoryUsed = gpu.memoryUsed
</code></pre>
<p>memory was still not 0, while torch.cuda.memory_allocated(0) shows 0.</p>
<p>how do you calculate the GPU memory that a pytorch model uses?<br>
or how do you compare the GPU memory that a pytorch model uses and its script-mode uses?</p>
<p>and if I understood right and used the right functions, why is the actual allocated memory that different from real torch tensor bytes?<br>
I knew it could be different because of using defined page sizes but I didn’t expect that it could be that much different (2GB difference).</p>

ptrblck: <p>PyTorch will create the CUDA context in the first CUDA operation, which will load the driver, kernels (native from PyTorch as well as used libraries etc.) and will take some memory overhead depending on the device.<br>
PyTorch doesn’t report this memory which is why <code>torch.cuda.memory_allocated()</code> could return a 0 allocation.<br>
You would thus need to use <code>nvidia-smi</code> (or any other “global” reporting tool) to check the overall GPU memory usage.</p>

ndvbd: <p>Is there a way to measure the peak gpu memory consumption between two points in time? For example, I am running a forward call on some network. Memory start at value x, then during the forward call goes to y, and then at the end back to x- I would like to discover the y value.</p>

WANGSSSSSSS: <p><code>torch.cuda.max_memory_allocated(*device=None* )</code> is all you need!<br>
<a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch.cuda.max_memory_allocated" rel="noopener nofollow ugc">torch.cuda.max_memory_allocated</a></p>

ndvbd: <p>Thank you.<br>
Strangely however, this does not yield 0:</p>
<pre><code class="lang-auto">torch.cuda.reset_peak_memory_stats(device=None)
print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
</code></pre>

ptrblck: <p>It works for me:</p>
<pre><code class="lang-python">print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
# gpu used 0 memory

x = torch.randn(1024, device="cuda")
y = torch.randn(1024, device="cuda")
print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
# gpu used 8192 memory

# should return same memory usage as nothing was deleted
torch.cuda.reset_peak_memory_stats(device=None)
print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
# gpu used 8192 memory

# delete tensors and reduce peak memory usage
del y
torch.cuda.reset_peak_memory_stats(device=None)
print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
# gpu used 4096 memory

del x
torch.cuda.reset_peak_memory_stats(device=None)
print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory")
# gpu used 0 memory
</code></pre>

ndvbd: <p>I’ve put it in a random place in my program, and it didn’t show 0. It should, right? There is occupied memory by the GPU, but if it’s right after the reset_peak, I guess it should show 0, but it doesn’t.</p>

ptrblck: <p>No, it shouldn’t since you are only resetting the peak which makes the currently used memory the new peak.</p>

ndvbd: <p>Well, that’s what is happening to me. strange</p>

ptrblck: <p>In case you misunderstood me: calling <code>reset_peak_memory</code> should <em>not</em> always show a 0 memory usage, since memory could still be allocated making it the new peak as seen in my code snippet.</p>

galen_pereira: <p>So how to convert that to bytes or MB etc ?</p>

ptrblck: <p>The returned values are given as bytes as described in the <a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html">docs</a>. You can simply convert them to MB by diving them by <code>1024**2</code>.</p>

Nancy76: <p>You need to know the size of each parameter (e.g., 4 bytes for float32) and estimate activation and optimizer states based on your model’s architecture. <a href="https://liteblue.cloud" rel="noopener nofollow ugc">Liteblue</a>  Tools like TensorBoard or built-in profiling tools in frameworks like PyTorch or TensorFlow can also help.</p>