Gradient is None initially

username: Tornike
cooked: <p>hi, im training Vits TTS. at some point me losses go to Nan.<br>
i set autograd.anomaly_detect and it return this error :</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 75, in _wrap
    fn(i, *args)
  File "/content/vits/vits/train_ms.py", line 133, in run
    train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler,
  File "/content/vits/vits/train_ms.py", line 234, in train_and_evaluate
    scaler.scale(loss_gen_all).backward()
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 525, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 267, in backward
    _engine_run_backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'ConvolutionBackward0' returned nan values in its 0th output.

</code></pre>
<p>is it oaky that my model has nan gradients initially ? what should I do?</p>
<p>anomaly detection turned off training at 0ht iteration</p>

username: KFrank
cooked: <p>Hi Tornike!</p>
<aside class="quote no-group" data-username="Tornike" data-post="1" data-topic="205541" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tornike/48/36047_2.png" class="avatar"> Tornike:</div>
<blockquote>
<p>is it oaky that my model has nan gradients initially ?</p>
</blockquote>
</aside>
<p>No.  Once some of your gradients become <code>nan</code>, your optimization step will<br>
cause the associated weights to become <code>nan</code>, causing more gradients to<br>
become <code>nan</code>, and so on.</p>
<aside class="quote no-group">
<blockquote>
<p>what should I do?</p>
</blockquote>
</aside>
<p>Get rid of the <code>nan</code>s.  First check that your input data doesn’t contain any<br>
<code>nan</code>s or <code>inf</code>s (or other outlandish values).</p>
<p>It’s unlikely, but also verify that your model’s weights aren’t somehow<br>
being initialized with <code>nan</code>s or <code>inf</code>s.</p>
<p>Then try training with plain-vanilla <code>SGD</code> with a very low learning rate.<br>
Sometimes training is unstable right at the beginning, after which you<br>
can turn the learning rate up to a more practical value (or switch to a<br>
“less stable” optimizer such as <code>Adam</code>).  The same goes for momentum.<br>
Even if you want to use a largish value for momentum such as <code>0.90</code> or<br>
<code>0.95</code>, you sometimes have to start training with no (or small) momentum.</p>
<p>(My intuition about why this happens is that the randomly-initialized weights<br>
of your model can be out of kilter – imagine them sitting on the side of a<br>
steep valley on the loss surface – leading to very large gradients that can<br>
cause you to jump to an even worse location and finally to <code>inf</code> or <code>nan</code><br>
gradients.  Training very slowly, i.e., small learning rate which means<br>
small optimizer step size, allows your weights to move down the loss<br>
surface, rather than, say, jump to the opposite side of the valley, to a<br>
more “sane” location that has more moderate gradients, at which point<br>
you can manage to increase the learning rate.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Tornike
cooked: <p>Hi Frank,</p>
<p>Thank you for your valuable reply.</p>
<p>i decided to go with fp32 and it worked. did not return any nan.</p>
<p>but it could not learn. after 48k steps my tts model has robotic voice.</p>
<p>losses almost did not decrase after 5-10k steps. except feature matching loss.</p>
<p>I’m training vits TTS.</p>
<p>any suggestions?</p>
<p>Thanks</p>