Why does model consume more memory on GPU than on storage

username: MaazJamal
cooked: <p>I wrote a simple bare bones program to check the usage of ram of gpu using pretrained resnet-34 from model zoo.  I found that the creating and shifting the model to GPU utilizes about 532 MBs of gpu ram. Loading the model seems to have no effect on ram usage, showing pytorch reserves the ram for subsequent loading of weights. The weight file is only 87.3 MBs on disk. So why is the model taking up 532 mb of GPU ram, even if we count class/function overhead why is it using over 5.5x the space it occupies on the disk? Is there anyway to reduce this usage?</p>

username: Stone
cooked: <p>Can you post how do you check the GPU RAM usage?<br>
Probably pytorch just cached the CUDA memory, for future allocation.</p>

username: MaazJamal
cooked: <p>Using nvidia-smi. I expanded my script to include output from pytorch own commands for checking memory.</p>
<pre><code class="lang-auto">import torch
from torchvision import models

a = models.resnet34(pretrained=False)
a.cuda()
a.load_state_dict(torch.load('./resnet34-333f7ec4.pth'))
a.eval()

print("Max Memory occupied by tensors: ", torch.cuda.max_memory_allocated(device=None))
print("Max Memory Cached: ", torch.cuda.max_memory_cached(device=None))
print("Current Memory occupied by tensors: ", torch.cuda.memory_allocated(device=None))
print("Current Memory cached occupied by tensors: ", torch.cuda.memory_cached(device=None))

input("press any key to continue")

b = torch.rand(1,3,224,224)
b = b.to('cuda')
c = a(b)

print("\n\nMax Memory occupied by tensors: ", torch.cuda.max_memory_allocated(device=None))
print("Max Memory Cached: ", torch.cuda.max_memory_cached(device=None))
print("Current Memory occupied by tensors: ", torch.cuda.memory_allocated(device=None))
print("Current Memory cached occupied by tensors: ", torch.cuda.memory_cached(device=None))

input("Press enter to exit")

</code></pre>
<p>Here is the output:<br>
Max Memory occupied by tensors:  87333888<br>
Max Memory Cached:  90570752<br>
Current Memory occupied by tensors:  87333888<br>
Current Memory cached occupied by tensors:  90570752<br>
press any key to continue</p>
<p>Max Memory occupied by tensors:  146021376<br>
Max Memory Cached:  156631040<br>
Current Memory occupied by tensors:  119782400<br>
Current Memory cached occupied by tensors:  156631040<br>
Press enter to exit</p>
<p>Im also attaching the output from nvidia-smi:<br>
Before passing an input<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/58e16c389eba4a6b2d0c8abab831a4196c4a6609.png" data-download-href="https://discuss.pytorch.org/uploads/default/58e16c389eba4a6b2d0c8abab831a4196c4a6609" title="Screenshot%20from%202019-05-28%2013-08-30"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/58e16c389eba4a6b2d0c8abab831a4196c4a6609_2_517x96.png" alt="Screenshot%20from%202019-05-28%2013-08-30" data-base62-sha1="cGgYY9rCcUofXazvJgL1qlgfRGx" width="517" height="96" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/58e16c389eba4a6b2d0c8abab831a4196c4a6609_2_517x96.png, https://discuss.pytorch.org/uploads/default/original/2X/5/58e16c389eba4a6b2d0c8abab831a4196c4a6609.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/58e16c389eba4a6b2d0c8abab831a4196c4a6609.png 2x" data-dominant-color="3C1831"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-05-28%2013-08-30</span><span class="informations">714×134 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
After passign an input:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1974d0d5479e8ba841b606f1477f6741dd068100.png" data-download-href="https://discuss.pytorch.org/uploads/default/1974d0d5479e8ba841b606f1477f6741dd068100" title="Screenshot%20from%202019-05-28%2013-09-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1974d0d5479e8ba841b606f1477f6741dd068100_2_517x102.png" alt="Screenshot%20from%202019-05-28%2013-09-20" data-base62-sha1="3Dcde9Q44vYVgnqEI6qTLHZaL3G" width="517" height="102" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1974d0d5479e8ba841b606f1477f6741dd068100_2_517x102.png, https://discuss.pytorch.org/uploads/default/original/2X/1/1974d0d5479e8ba841b606f1477f6741dd068100.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/1974d0d5479e8ba841b606f1477f6741dd068100.png 2x" data-dominant-color="3B1830"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-05-28%2013-09-20</span><span class="informations">715×141 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is still a huge discrepancy between max memory cached and the amount of memory occupied as shown by nvidia-smi. Am i using the functions incorrectly?</p>

username: Stone
cooked: <p>Pytorch uses some memory for meta-data and CUDA context management, and it’s not exposed to python API yet.</p>
<p>You can try:</p>
<pre><code class="lang-python">a = torch.Tensor(1).cuda()
</code></pre>
<p>And then <code>nvidia-smi</code> to see how much extra memory pytorch requires.</p>