The min_scale should be set for AMP.GRAD_SCALER

username: smartbarbarian
cooked: <p>The scale must be greater than 1.0, otherwise it will increase gradient underflow. However, due to instability during training, the scale may drop below 1.0, causing a death spiral and ultimately crashing the entire training process.</p>

username: ptrblck
cooked: <p>You are right that instabilities during training could cause a “death spiral”.<br>
However, I would rather recommend to fix the actual training instead of trying to limit e.g. the scaling factor as it also won’t save you from the actual failure.</p>

username: smartbarbarian
cooked: <p>Based on my testing with the model, a high total gradient norm may cause the loss scale to decrease progressively, potentially leading to a death spiral. As a solution, I have manually set the minimum scale to 1.0. Given enough time, the training process tends to stabilize, and the scale usually maintains a normal value of around 32.0. It would be beneficial to allow these developing models an opportunity to adjust their initial parameters.</p>
<p>Additionally, it is recommended that the scaler update function omits batches with NaN losses. My model occasionally produces NaN values due to mathematical operations, although this happens only 0.5 times per 1000 batches. Nonetheless, this could contribute to a death spiral as well.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="smartbarbarian" data-post="3" data-topic="178447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b4bc9f/48.png" class="avatar"> smartbarbarian:</div>
<blockquote>
<p>Additionally, it is recommended that the scaler update function omits batches with NaN losses. My model occasionally produces NaN values due to mathematical operations, although this happens only 0.5 times per 1000 batches. Nonetheless, this could contribute to a death spiral as well.</p>
</blockquote>
</aside>
<p>Again, I would recommend trying to fix the actual NaN outputs or to skip the iteration manually in case these NaN outputs are created as it sounds as if you are trying to change the grad scaling util. to fix an already unstable training.<br>
I would be interested to learn more about the high gradient magnitudes which are not caused by a huge loss or invalid outputs. I.e. in particular which operations seems to blow up the gradients causing the overflow during the scaling.</p>

username: smartbarbarian
cooked: <ol>
<li>
<p>In my model, I employed torch.linalg.solve and torch.det, which can result in NaN values when the positive-definite condition is violated due to underflow.</p>
</li>
<li>
<p>The total gradient norm of my model is in the hundreds, and occasionally, it reaches the thousands. However, the loss and outputs seem valid, perhaps because I used torch.nn.utils.clip_grad_norm_, and my model functions well. I am curious to know if it is typical to have such a large total gradient norm.</p>
</li>
</ol>

username: ptrblck
cooked: <aside class="quote no-group" data-username="smartbarbarian" data-post="5" data-topic="178447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b4bc9f/48.png" class="avatar"> smartbarbarian:</div>
<blockquote>
<p>The total gradient norm of my model is in the hundreds, and occasionally, it reaches the thousands. However, the loss and outputs seem valid, perhaps because I used torch.nn.utils.clip_grad_norm_, and my model functions well.</p>
</blockquote>
</aside>
<p>I don’t fully understand this explanation. Is the gradient norm in the thousands before or after unscaling?<br>
If afterwards, how would a low scaling factor cause an underflow? I also assume you are clipping the gradients after unscaling as described in <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-clipping">this amp example</a>?</p>
<aside class="quote no-group" data-username="smartbarbarian" data-post="5" data-topic="178447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b4bc9f/48.png" class="avatar"> smartbarbarian:</div>
<blockquote>
<p>I am curious to know if it is typical to have such a large total gradient norm.</p>
</blockquote>
</aside>
<p>I don’t know as I haven’t seen it before, but I’m also not familiar with your use case as you are already expecting to see NaN losses sometimes, which is also uncommon.</p>

username: smartbarbarian
cooked: <p>It’s the gradient norm in the thousands after unscaling.<br>
I presume that in my model, the gradient norm oscillates, frequently soaring high and then plummeting low. The loss scale tends to decrease with a high gradient norm. However, when the gradient norm falls to a low level, the loss scale struggles to promptly recover to a higher value.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c5054ac129f3d3698906d12bff9c221b93f4730.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c5054ac129f3d3698906d12bff9c221b93f4730" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c5054ac129f3d3698906d12bff9c221b93f4730_2_325x500.png" alt="image" data-base62-sha1="daE2b0gfHRgk3h4ezD62Ftx4UIo" width="325" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c5054ac129f3d3698906d12bff9c221b93f4730_2_325x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c5054ac129f3d3698906d12bff9c221b93f4730.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c5054ac129f3d3698906d12bff9c221b93f4730.png 2x" data-dominant-color="E1E0DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">354×544 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>