Batchnorm2d outputs NaN - Negative running_var

ragnaarok: <p>Hi,</p>
<p>I’m trying to understand and solve a problem where my loss goes to nan. Information I have:</p>
<ul>
<li>Fp16 training (autocast, scale().backward, unscale, clip_grad_norm, scaler.step, scaler.update, zerograd) diverges to Nan</li>
<li>I found the issue in a batchnorm layer during an fp32 inference</li>
<li>It goes: convolution2d &gt; x &gt; batchnorm2d &gt; some feature maps are full of NaN</li>
</ul>
<p>After checking in depth (manually recomputing batchnorm), I found that some of my batchnorm.running_var have negative values which isn’t supposed to happen (it produces nan when the batchnorm applies sqrt() on these values).</p>
<p><strong>How could the values in running_var be negative?</strong></p>
<p>Does it mean that there’s an issue with the scaler during the training?</p>
<p>Thanks to anyone with more information on this issue</p>

ptrblck: <p>This is indeed strange. Did you see the negative var values after the mixed-precision training when you tried to evaluate the model in FP32?<br>
I assume you didn’t set a negative <code>momentum</code> or anything like that?</p>

ragnaarok: <p>Momentum was 0.1 and eps was 1e-5.<br>
The negative var values appear during the training so I guess that even with the regular fp16 training procedure there are still ways to have an overflow here.</p>
<p>After I changed my model I have more stable trainings. I torch.clamp the output and the input of the model, I also apply tanh after each convolution such that having big values here or in the batchnorm would be useless.</p>
<p>Apart from an overflow I don’t see how I could get a negative variance in the batchnorm</p>

ptrblck: <p>Batchnorm layers are running in FP32 in an <code>autocast</code> region, so they shouldn’t overflow internally.<br>
Just to make sure I’m understanding your use case: “regular fp16 training” means you are using <code>torch.cuda.amp</code> or are you calling manually <code>half()</code> on any modules?</p>

ragnaarok: <p>No no I’m not doing half() anywhere, It means I’m using the procedure from here: <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision examples — PyTorch 1.8.0 documentation</a>.</p>
<p>My code:</p>
<pre><code class="lang-auto">Main while loop:
    fetching batch from dataloader
    batch['x'] = batch['x'].cuda(non_blocking=True)

    #infer
    with torch.cuda.amp.autocast():
        out = model(batch['x'])
        loss = loss_model(out, batch['x']) #autoencoder loss
    
    #backprop
    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5) 
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()
</code></pre>
<p>batch[‘x’] is of shape [64, 1, 256, 150] with values ranging from 0 to 1e4.<br>
I know the troubles with manually using half() on a model</p>

tom: <p>Which batch norm are you getting (i.e. what is the grad_fn of the batch norm output, it could be cudnn or native)?</p>

ragnaarok: <p>The grad_fn for the output just after the batchnorm is CudnnBatchNormBackward</p>

tom: <p>OK. Then it’s at least not my code doing it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> I had tried to see if I could get the Welford variance update to be negative, but it seems to not be easy (you would need <code>x - avg</code> and <code>x-(avg + (x-avg)/n)</code> to be of different signs).</p>
<p>More seriously, you could try to see if you can avoid the problem by calling batch norm in a <code>with torch.backends.cudnn.flags(enabled=False):</code> block (but I didn’t try if it works or helps).</p>
<p>Best regards</p>
<p>Thomas</p>

ragnaarok: <p>Thanks, I’ll try that if I have this error again.</p>
<p>As I said, It seems that my training is more stable after I changed some tweaks in my model so I guess I just had an unstable architecture.<br>
It’s weird that a variance could be negative so maybe it’ll be useful to fix that if someone else have the same problem later or if it doesn’t make sense in the existing code.</p>
<p>My architecture was a convolutional autencoder, 2dmap &gt; conv2d(bn2d/act) &gt; conv2d &gt; … &gt; avgpool2d &gt; features &gt; bn1d &gt; linear &gt; bn1d &gt; convT2d &gt; … &gt; 2dmap. Removing the two 1D-BN in the middle maybe solved my issue, but I’m not sure and of course it’s weird because the negative variances are in the first 2D-BN (I’m leaving that here in case someone else have the same issue later)</p>

ptrblck: <aside class="quote no-group" data-username="ragnaarok" data-post="9" data-topic="115373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e19adc/48.png" class="avatar"> ragnaarok:</div>
<blockquote>
<p>As I said, It seems that my training is more stable after I changed some tweaks in my model so I guess I just had an unstable architecture.</p>
</blockquote>
</aside>
<p>I don’t think the error was caused by the architecture, but as <a class="mention" href="/u/tom">@tom</a> mentioned, might be a faulty kernel in cudnn, which should be fixed. In case you have an executable code snippet or run into this error again, could you please post it here?</p>

jxtps: <p>I’m running into this issue as well. I’ve debugged it a bit by dumping the input data &amp; network <code>state_dict</code> as soon as a <code>NaN</code> loss is detected.</p>
<p>When I step through the network with that data a specific <code>Conv2d</code> produces an output that has some <code>inf</code> elements. The <code>Conv2d</code>’s weights &amp; bias appear reasonable:</p>
<pre><code class="lang-auto"> |   mean   |   min    |   max    |   std    |       shape        || param_name  
 |    0.208 |   -1.560 |    9.680 |    0.997 |    [64, 128, 3, 3] || weight
 |    0.662 |   -2.702 |    8.658 |    1.713 |               [64] || bias
</code></pre>
<p>My guess is that the <code>inf</code> elements lead to <code>NaN</code>s when incorporated into the <code>BatchNorm2d</code>’s running stats.</p>
<p>I have dumped &amp; can share the specific inputs &amp; state dicts for the <code>Conv2d</code> &amp; <code>BatchNorm2d</code> if they would be of any help in debugging this, but the <code>BatchNorm2d</code> has already been polluted and has a <code>NaN</code> entry in each of <code>running_mean</code> and <code>running_var</code> - seems like there’s first an <code>inf</code> loss, then comes the <code>NaN</code>, though I’ve not confirmed that hypothesis.</p>
<p>Ideally the BatchNorm2d layer would not update its running stats in AMP training with <code>inf</code> inputs, so that the scaler can fully roll back the iteration when <code>inf</code> is detected.</p>
<p>(Maybe it already does that, and the issue is more complicated, idk)</p>

jxtps: <p>Confirmed that BatchNorm2d’s running stats do indeed get ruined when it receives <code>inf</code> input.</p>
<p>Filed a bug for this: <a href="https://github.com/pytorch/pytorch/issues/90342" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm2d doesn't handle inf input in AMP training · Issue #90342 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>Invalid activation values during the forward pass are not expected during mixed-precision training as explained in your created <a href="https://github.com/pytorch/pytorch/issues/90342#issuecomment-1340240936">issue</a> and you would thus need to make sure your training isn’t exploding. The <code>GradScaler</code> is responsible for gradient scaling and cannot rewind the forward pass.</p>

vadimkantorov: <p>invalid activation values can happen sometimes. unfortunately, there is no protection against updates of running-stats against bad values, and no way to cancel the update. what might be a good pattern is to e.g. cache previous iteration of running_stats and have a method to recover them (e.g. if the loss is large and we’d like to skip the batch)</p>