GPU running out of memory

username: Izu97
cooked: <p>I try to run CNN model on GPU with the input shape of (3,224,224) .It occur the following issues . Here is the nvidia-smi output. How I can free up the GPU memory. Thank you.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd048cce734ca3d87e7db0040cec89f18a642bdc.png" alt="nvidia_smi" data-base62-sha1="A6iy04T2VjdE4WjQsED9MuhtevW" width="635" height="253"></p>
<p>Error Msg:<br>
data. defaultcpuallocator: not enough memory: you tried to allocate 34798181769216 bytes. buy new ram!</p>

username: ptrblck
cooked: <p>The error message points to your system RAM, not the GPU memory.<br>
It seems you are trying to create a huge tensor on the CPU.<br>
Could you post the line of code, which raises this issue?</p>

username: braindotai
cooked: <p>One simple solution is you can lower the batch size until everything works.<br>
As what <a class="mention" href="/u/ptrblck">@ptrblck</a> has said its a cpu allocation issue, try using gpu by calling <code>.cuda()</code> to your model and dataset.<br>
And if you still get error in this case by using gpu then try freeing memory allocated at gpu using <code>torch.cuda.empty_cache()</code> after every epoch or batch iteration.<br>
Other wise I’d recommend you using gradient accumulation, more about it <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">here</a>, using this you can your train model using data with bigger batch size even if your gpu doesn’t have that much memory.</p>

username: ptrblck
cooked: <p>If the number of correctly calculated, this would result in 35TB, which seems to be quite high and I guess the code might have some bug/typo somewhere. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Izu97
cooked: <p>I just used  <strong>.to(device)</strong> method to assign tensors to GPU.</p>

username: Izu97
cooked: <p>Now I got this error:<br>
RuntimeError: CUDA out of memory. Tried to allocate 588.00 MiB (GPU 0; 4.00 GiB total capacity; 2.95 GiB already allocated; 150.76 MiB free; 2.97 GiB reserved in total by PyTorch)</p>
<p>and optimizer.step() generate the error.</p>

username: braindotai
cooked: <p>What is your batch size? I think its too high for your gpu to allocate to its memory. As I said use gradient accumulation to train your model.<br>
If you want to train with batch size of <code>desired_batch_size</code>, then divide it by a reasonable number like 4 or 8 or 16…, this number is know as <code>accumtulation_steps</code>. Now change your batch size for the dataset to <code>desired_batch_size/accumulation_steps</code> and train your model as below</p>
<pre><code class="lang-python">for epoch in range(epochs):
    for i, (inputs, labels) in enumerate(training_set):
        predictions = model(inputs)                     # Forward pass
        loss = loss_function(predictions, labels)       # Compute loss function
        loss = loss / accumulation_steps                # Normalize our loss (if averaged)
        loss.backward()                                 # Backward pass
        if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
            optimizer.step()                            # Now we can do an optimizer step
            model.zero_grad()                           # Reset gradients tensors
</code></pre>