Incorrect MSE loss for float16

Anandh_Perumal_Konar: <pre><code class="lang-auto">loss_function = MSELoss()
loss_function(torch.tensor([0.0329]).to(torch.float16), torch.tensor([60000]).to(torch.float16))
--&gt; tensor(inf, dtype=torch.float16)
</code></pre>
<p>why is the results inf?</p>

ptrblck: <p><code>float16</code> has a max range of <code>+- 65504</code> and will overflow to <code>+- Inf</code> outside of this range.<br>
It’s thus expected that <code>nn.MSELoss</code> will overflow via <code>(0.03 - 60000)**2 ~= 3.6e9</code></p>

Anandh_Perumal_Konar: <p>Thank you for input. so its not possible to train a fp16 model for mse, since it’s going to be inf in most of the cases when loss is higher than 65k</p>

ptrblck: <p>Training any model in pure <code>float16</code> is tricky as not only will large activation values potentially overflow but your training would also suffer from underflowing small gradients.<br>
This is why we’ve developed the mixed-precision training util. via <code>torch.amp</code>, which not only uses an <code>autocast</code> context to transform tensors to <code>float16</code> when it’s safe, but also uses a loss scaler to avoid underflows. Take a look at the <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html">AMP recipe</a> and the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">examples</a> to see how to use it.</p>

Anandh_Perumal_Konar: <p>I tried that but I get error the following error</p>
<pre><code class="lang-auto">RuntimeError: Found dtype Float but expected Half</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce the error, please?</p>

Anandh_Perumal_Konar: <p>Can you tell me what part of code would be helpful? Since, it is very modular, now sure I can paste the whole code.</p>

ptrblck: <p>The first link already does it as it describes a simple network first with a standard training loop in default precision. In the next section <code>autocast</code> is added and afterwards the <code>GradScaler</code> both with code changes and with explanations why these utils. are used. Then the same initial code is posted again as <code>All together: "Automatic Mixed Precision"</code>. Did you walk through this doc and got stuck somewhere?</p>

Anandh_Perumal_Konar: <p>Yes, I went through the doc. I think now the code works for me but I get error while updating the learning rate scheduler. is this the correct way to update learning rate scheduler</p>
<pre><code class="lang-auto">scaler.step(self.lr_scheduler)
</code></pre>
<p>or should I call it in the conventional way of self.lr_scheduler.step()<br>
where as self.lr_scheduler.step() works for me.<br>
I get the following error</p>
<pre><code class="lang-auto">'LambdaLR' object has no attribute 'param_groups'</code></pre>

ptrblck: <p><code>scaler.step</code> expects an optimizer, so use <code>lr_scheduler.step()</code> instead.</p>

Anandh_Perumal_Konar: <p>For amp, my model is predicting inf. I’m getting inf only during amp training and not during full precision training. Because of inf I’m getting error in sklearn during accuracy computation during training.<br>
Any thoughts, why this is happening?</p>

ptrblck: <p>The forward method should not overflow so I don’t know what might be causing it and would need to get more information about the model etc.</p>

Anandh_Perumal_Konar: <p>If you look at this part of the code from the doc:</p>
<pre><code class="lang-auto">        with torch.autocast(device_type='cuda', dtype=torch.float16, enabled=use_amp):
            output = net(input)
aren't we telling the model output should always be float16? which will cause the overflow?</code></pre>