akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA out of memory even the parameters is less than before

fllci: <p>Hi all!</p>
<p>I’ve trained my 80million parameter -model successfully on the same system.</p>
<p>Few weeks later, I decreased my model network to 18 million parameter but now it is saying that</p>
<blockquote>
<p>CUDA out of memory</p>
</blockquote>
<p>How is this even possible?</p>
<p>Thanks!</p>

Andrei_Cristea: <p>Hi Furkan,</p>
<p>Two potential culprits jump to mind here:</p>
<ol>
<li>Your system is running some other GPU-consuming process. Run <a href="https://developer.nvidia.com/nvidia-system-management-interface" rel="noopener nofollow ugc">nvidia-smi</a> to rule this out.</li>
<li>You’ve introduced a bug into your training that is causing your script to do something additional that’s memory-consuming. An example of this would be to keep running track of your losses by adding the tensor loss from each training batch, rather than adding the <em>value</em> of that loss via the .item() method. This would cause you to store in memory an expanding and GPU-consuming computation graph.</li>
</ol>
<p>Without more detail, it’s hard to give a clear answer, but perhaps you could investigate those two hypotheses and see if anything pops up.</p>
<p>Best,<br>
Andrei</p>

fllci: <aside class="quote no-group" data-username="Andrei_Cristea" data-post="2" data-topic="168410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>Without more detail, it’s hard to give a clear answer, but perhaps you could investigate those two hypotheses and see if anything pops up.</p>
</blockquote>
</aside>
<p>Thanks Andrei. Your suggestions are probably the reason of this issue. I actually updated the torch to 1.13.1. Then, the model was run without a hitch.<br>
Best,<br>
Furkan</p>
Passing PyTorch Batch to HuggingFace Wav2vce2 Model

shakeel608: <p>I have a PyTorch batch of size say (2, 1, 100)<br>
and I want to pass this batch to HuggingFace Wav2Vec2 Model, first to feature_extractor and then to base wav2vec2 model</p>
<ol>
<li>input in forward function is x of shape (2,1,100)<br>
What I want in forward function is like this</li>
</ol>
<pre><code class="lang-auto">x_enc = feature_extractor(x sampling_rate=16000)
out = wav2vec2_model(x)
</code></pre>
<p>It might seems naive, but any help would be appreciated</p>
<p>Does anyone have any idea how to make it compatible with Hugging Face</p>
<p>Thank you</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>