Where is all the memory going?

username: Simon_Berrebi
cooked: <p>The following error message is confusing. If I have 22Gb of total capacity and only 6 Mb free, how can I check where the rest is going?</p>
<p><code>OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 21.99 GiB total capacity; 1.04 GiB already allocated; 6.12 MiB free; 1.18 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>Here is my code:</p>
<pre><code class="lang-auto">model = BertModel.from_pretrained('bert-base-uncased')

# Check for GPU availability and set the device accordingly
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = model.to(device)

# Cleanup
model.eval()

batch_size = 50
num_batches = len(tokenized_dataset["input_ids"]) // batch_size + (1 if len(tokenized_dataset["input_ids"]) % batch_size != 0 else 0)

# Create an empty list to store the output
results = []

# Forward pass
with torch.no_grad(): 
    for i in range(num_batches):
        start = i * batch_size
        end = start + batch_size
        
        input_ids = torch.tensor(tokenized_dataset["input_ids"])[start: end].to(device)
        attention_mask = torch.tensor(tokenized_dataset["attention_mask"])[start: end].to(device)

        # feed the batch into the model
        output = model(input_ids, attention_mask= attention_mask)

        # Try to free up some memory
        del input_ids
        del attention_mask
        torch.cuda.empty_cache() 

        # appending the output to the results list
        results.append(output)
</code></pre>

username: Soumya_Kundu
cooked: <p>Struggling a bit to understand your question. Are you asking about the remaining 6mb or where the 22gb is going?</p>
<p>To monitor your vram usage in real time, nvitop is a good option.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/XuehaiPan/nvitop">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/XuehaiPan/nvitop" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/436;"><img src="https://repository-images.githubusercontent.com/332736941/9246d53d-8d5f-4769-8a29-d70f924ef031" class="thumbnail" width="690" height="436"></div>

<h3><a href="https://github.com/XuehaiPan/nvitop" target="_blank" rel="noopener nofollow ugc">GitHub - XuehaiPan/nvitop: An interactive NVIDIA-GPU process viewer and...</a></h3>

  <p>An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management. - XuehaiPan/nvitop</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>PyTorch profilers are obviously better but will need a bit of time investment.</p>

username: Simon_Berrebi
cooked: <p>Thanks thats helpful!</p>
<p>I would expect 24.00 MiB not to be a problem because I have 22 Gb total capacity. <code>already allocated + free + reserved in total by PyTorch</code> only account for 2 Gb.</p>
<p>My question is: what is happening with the remaining 20 Gb and why can’t I use them to model inference?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Simon_Berrebi" data-post="1" data-topic="208799">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simon_berrebi/48/62378_2.png" class="avatar"> Simon_Berrebi:</div>
<blockquote>
<p>6.12 MiB free</p>
</blockquote>
</aside>
<p>Check other processes and the general GPU utilization in <code>nvidia-smi</code> to see who is using the memory.</p>