RuntimeError: CUDA out of memory. Tried to allocate - Can I solve this problem?

FurkanGozukara: <p>Hello everyone. I am trying to make CUDA work on open AI whisper release. My current setup works just fine with CPU and I use medium.en model</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/openai/whisper/">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/openai/whisper/" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/8463579b39288189e7480079202587765d4fe2b60261fbe68debf51ddb15c6bf/openai/whisper" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/openai/whisper/" target="_blank" rel="noopener nofollow ugc">GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak...</a></h3>

  <p>Robust Speech Recognition via Large-Scale Weak Supervision - GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have installed CUDA-enabled Pytorch on Windows 10 computer however when I try speech-to-text decoding with CUDA enabled it fails due to ram error</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 70.00 MiB (GPU 0; 4.00 GiB total capacity; 2.87 GiB already allocated; 0 bytes free; 2.88 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>I disabled and enabled the graphic card before running the code - thus the VGA ram was 100% empty. Tested on my laptop so it has another GPU as well. Therefore, my GTX 1050 was literally using 0 MB of memory already</p>
<p>My computer also has 32 GB ram and CPU synthesis working very well but just too slow. In 7 hours processed only 1 hour of speech.</p>
<p><strong>My installed version :</strong></p>
<p>Name: torch<br>
Version: 1.12.1+cu116<br>
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration<br>
Home-page: pytorch org<br>
Author: PyTorch Team<br>
Author-email: packages pytorch org<br>
License: BSD-3<br>
Location: c:\python399\lib\site-packages<br>
Requires: typing-extensions<br>
Required-by: whisper, torchvision, torchaudio</p>
<p><strong>Do you have any suggestions to solve this problem?</strong></p>
<p><a href="https://i.imgur.com/2TRloGX.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/2TRloGX.png</a></p>

tom: <p>The problem could be the GPU memory used from loading all the Kernels PyTorch comes with taking a good chunk of memory, you can try that by loading PyTorch and generating a small CUDA tensor and then check how much memory it uses vs. how much PyTorch says it has allocated.<br>
There has been work to put PyTorch on a bit of a diet there (e.g. the “JITerator”) but I’m not sure about the state, in particular on Windows.</p>
<p>Fun fact: In the olden times, PyTorch would print “Buy more RAM” along with the error message, but then things got all serious. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Best regards</p>
<p>Thomas</p>

FurkanGozukara: <p>Could you give me instructions to run and see? I really don’t know how to do what you are saying <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> Can we assign somehow virtual ram to the GPU? I am ok with it being slower than dedicated GPU ram</p>

FurkanGozukara: <p>I started considering purchasing 12 GB RTX 3060 card what you think?</p>

tom: <p>Start Python and do <code>import torch ; a = torch.ones(1, device="cuda")</code> or so. And then check the GPU memory usage (it’s <code>nvidia-smi</code> on Linux, I don’t quite know on windows).</p>
<p>Best regards</p>
<p>Thomas</p>

FurkanGozukara: <p>Thank you very much.</p>
<p>Also I just ordered 12 gb RTX 3060 (411$ in Turkey almost equal to my monthly salary)</p>
<p>I hope it can run big model</p>

tom: <p>These GPUs are decidedly not cheap. I feel for you!</p>

Cater_Wang: <p>Hey, bro. Did you solve this problem after changing your GPU to the one with more memory？</p>

FurkanGozukara: <p>After GPU change it works great<br>
I am daily using it<br>
Here my video : <a href="https://youtu.be/msj3wuYf3d8" class="inline-onebox" rel="noopener nofollow ugc">How to do Free Speech-to-Text Transcription Better Than Google Premium API with OpenAI Whisper Model - YouTube</a></p>

SM2023: <p>Hi <a class="mention" href="/u/furkangozukara">@FurkanGozukara</a>, how did you solve the issue of the Cuda memory?</p>

AlexPall: <p>To solve the “CUDA out of memory” error, reduce the batch size, use a smaller model, clear unnecessary variables with <code>torch.cuda.empty_cache()</code>, or upgrade your GPU for more memory.</p>

Raymond_Cool: <p>Neither this nor manual garbage collection resolves this in every case. I also see a lot of folks suggesting to decres\ase the batch size. This often does nothing either, I’m not the only one. Secondly, I’ve used the model I’m testing quite a bit, when this issue suddenly cropped up.</p>
<p>I’m not saying you shouldn’t share these steps, but they are in no way a guarantee. Something is amiss with the dependencies, or container, or both.</p>

AKQuestSage: <h2><a name="i-am-having-similar-issue-i-have-a-12gb-rtx-3060-but-it-tells-me-that-i-have-000gb-allocated-i-just-executed-my-working-code-which-works-from-a-different-gpu-and-machine-but-it-does-not-work-on-this-new-gpu-and-machine-1" class="anchor" href="#i-am-having-similar-issue-i-have-a-12gb-rtx-3060-but-it-tells-me-that-i-have-000gb-allocated-i-just-executed-my-working-code-which-works-from-a-different-gpu-and-machine-but-it-does-not-work-on-this-new-gpu-and-machine-1"></a>I am having similar issue. I have a 12gb RTX 3060 but it tells me that I have 0.00GB allocated.<br>
I just executed my working code which works from a different GPU and machine. But it does not work on this new GPU and machine.</h2>
<h2><a name="torchoutofmemoryerror-cuda-out-of-memory-tried-to-allocate-146-gib-gpu-0-has-a-total-capacity-of-1200-gib-of-which-1098-gib-is-free-of-the-allocated-memory-0-bytes-is-allocated-by-pytorch-and-0-bytes-is-reserved-by-pytorch-but-unallocated-if-reserved-but-unallocated-memory-is-large-try-setting-pytorch_cuda_alloc_confexpandable_segmentstrue-to-avoid-fragmentation-2" class="anchor" href="#torchoutofmemoryerror-cuda-out-of-memory-tried-to-allocate-146-gib-gpu-0-has-a-total-capacity-of-1200-gib-of-which-1098-gib-is-free-of-the-allocated-memory-0-bytes-is-allocated-by-pytorch-and-0-bytes-is-reserved-by-pytorch-but-unallocated-if-reserved-but-unallocated-memory-is-large-try-setting-pytorch_cuda_alloc_confexpandable_segmentstrue-to-avoid-fragmentation-2"></a>torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.46 GiB. GPU 0 has a total capacity of 12.00 GiB of which 10.98 GiB is free. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.</h2>

AKQuestSage: <p>Update: I also found a problem with the GPU during a different test. I’m not sure how it relates to this issue yet, but when I switched to a new and different GPU brand with the same 3060 12GB, everything worked. I am unable to continue figuring out what went wrong for I have already replaced my GPU.</p>
<p>It still shows an out-of-memory error when I intentionally over-allocate memory, but that’s expected. This is different from the reported issue where allocating only 1.46GB failed.</p>