Cuda out of memory with plenty left

username: Sagivfer
cooked: <p>Hey, I’ve created a detailed issue around the topic on git, moving it to here.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/119547">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/119547" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/119547" target="_blank" rel="noopener nofollow ugc">CUDA Memory error while having a lot of free memory</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-02-09" data-time="13:52:27" data-timezone="UTC">01:52PM - 09 Feb 24 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/Sagivfer" target="_blank" rel="noopener nofollow ugc">
          <img alt="Sagivfer" src="https://avatars.githubusercontent.com/u/61480454?v=4" class="onebox-avatar-inline" width="20" height="20">
          Sagivfer
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: memory usage
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: regression
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Hey,

I'm training a reinforcement learning agent on m<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">y device.
The models are rather small and the entire memory it takes (at peak usage) is around 870 MB.
My GPU is NVIDIA GeForce GTX 1650 TI with 4 GB memory.

I'm using CUDA V12.3
Torch V2.2.0 + cu118

My batch size is practically 1 since I'm iteratively sampling a transition, calculating the loss for it, and finally back propagating after accumulating enough transitions.
After processing the transition I'm moving it back to the CPU. 

I can't understand  why I'm getting the errors if the memory doesn't even get close to the capacity.

here are the errors that I'm getting and a snippet of the code that handles the loss calculations:
![error](https://github.com/pytorch/pytorch/assets/61480454/d31206de-c1a9-4c06-a0fe-aae18b89ac47)

![update_batch](https://github.com/pytorch/pytorch/assets/61480454/c4f56717-85af-4d2b-a1be-6d26badee025)

![update_loss](https://github.com/pytorch/pytorch/assets/61480454/61327b40-2cf0-4441-9ea8-e11cca1a68cb)

![backward](https://github.com/pytorch/pytorch/assets/61480454/015f597a-9c83-458f-ae82-78947ed467a6)

![parameters_update](https://github.com/pytorch/pytorch/assets/61480454/02c9a839-3bcd-443a-993f-abdd3a8e3660)

''' Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 4.00 GiB of which 2.35 GiB is free. Of the allocated memory 825.15 MiB is allocated by PyTorch, and 24.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management '''

### Versions

Collecting environment information...
PyTorch version: 2.2.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A
OS: Microsoft Windows 11 Home
GCC version: Could not collect
Clang version: 14.0.0
CMake version: Could not collect
Libc version: N/A
Python version: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22000-SP0
Is CUDA available: True
CUDA runtime version: 12.3.107
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650 Ti
Nvidia driver version: 546.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Revision=
Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-lightning==2.1.3
[pip3] torch==2.2.0+cu118
[pip3] torchaudio==2.2.0+cu118
[pip3] torchmetrics==1.3.0.post0
[pip3] torchrl==0.2.1
[pip3] torchvision==0.17.0+cu118
[conda] Could not collect

cc @ptrblck</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: vmoens
cooked: <p>Could be an issue with torchrl</p>
<p>If that comes from torchrl: can you tell us more about your replay buffer, specifically how you build it?<br>
What other components of torchrl / tensordict are you using?<br>
Does it replicate if you upgrade to torchrl 0.3.0?</p>

username: Sagivfer
cooked: <p>I’m using a custom implemented PER <a href="https://github.com/rlcode/per" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rlcode/per: Prioritized Experience Replay (PER) implementation in PyTorch</a></p>
<p>I’ve made sure to delete the tensors from memory in the implementation.</p>

username: vmoens
cooked: <p>Alright, not sure what you use from tensordict/torchrl in the code then. If you think the problem can come from there feel free to ping me</p>