Use global statistics of BatchNorm in training mode

username: deepak242424
cooked: <p>I am working on a edge detection code and my batch_size=1 (due to GPU RAM’s limitation). I am using a pretrained ResNet as my backbone architecture. While training I want to use the global statistics of batch norm layer rather than batch statistics (as it won’t make much sense because of batch_size=1).</p>
<p>One option is to set model.eval() in my training part so the network will use the global statistics of the batch norm layer. Given my network doesn’t have any dropout layers, is it the right thing to do?</p>
<p>What if my network has dropout layers and I want to use global batch norm statistics of bath norm layer in training phase?</p>

username: ptrblck
cooked: <p>It should work by setting <code>model.eval()</code>, if you don’t have any other modules, which switch their behavior in train/eval.<br>
However, a cleaner approach would be to just set the <code>BatchNorm</code> layers to eval.<br>
If you have a model definition, something like this should work:</p>
<pre><code class="lang-python">model = MyModel()
model.bn1.eval()
</code></pre>

username: deepak242424
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your reply.</p>
<p>I don’t have any other modules, which switch their behavior in train/eval and set  <code>model.eval()</code> in training phase. Now I am seeing Nan values in loss. However this is not the case when I set <code>model.train()</code> in my code.</p>
<p>Further I also tried to set individual batch norm layers to <code>eval()</code> mode (following your code snippet). Again I am observing <code>Nan</code> values.</p>
<p>Can you suggest anything that might be possibly going wrong?</p>

username: ptrblck
cooked: <p>That sounds strange. Can you check your input for NaN values with <code>print((x!=x).any())</code>?</p>

username: deepak242424
cooked: <p>I ran the code with model.eval(). After the last batch norm layer all the values are becoming Nan.However when I set model.train() the code is working fine.</p>
<p>Further I saw, the weights and bias parameters of the Batch Norm layers are going to Nan. This seems quite a weird behaviour.</p>
<p>At the following threads also people are trying to do the exact same thing. I followed all the solutions mentioned there but still no luck.</p>
<aside class="quote" data-post="4" data-topic="13214">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gurkirt/48/72167_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/proper-way-of-fixing-batchnorm-layers-during-training/13214/4">Proper way of fixing batchnorm layers during training</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/maunz">@maunz</a> 
I had same problem. It seems I setting eval mode wrong. 
This thread helped me <a href="https://discuss.pytorch.org/t/freeze-batchnorm-layer-lead-to-nan/8385">Freeze BatchNorm layer lead to NaN</a> 
def set_bn_to_eval(m): 
classname = m.class.name 
if classname.find(‘BatchNorm’) != -1: 
m.eval() 
net.apply(set_bn_to_eval) 
if you want to set_bn_to_eval of some subnet or some basenetwork then just use. 
net..apply(set_bn_to_eval) 
hope this helps.
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="1" data-topic="8385">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/6bbea6/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/freeze-batchnorm-layer-lead-to-nan/8385">Freeze BatchNorm layer lead to NaN</a> 
  </div>
  <blockquote>
    Hi, everyone 
I want to freeze BatchNorm while fine-tuning my resnet (I mean, use global mean/std and freeze weight and bias in BN), but the loss is so large and become nan at last: 
iter =  0 of 20000 completed, loss =  [ 15156.56640625]
iter =  1 of 20000 completed, loss =  [ nan]
iter =  2 of 20000 completed, loss =  [ nan]

the code I used to freeze BatchNorm is: 
def freeze_bn(model):
    for name, module in model.named_children():
        if isinstance(module, nn.BatchNorm2d):
            …
  </blockquote>
</aside>


username: Pavan_Teja
cooked: <p>Hey Its too late to ask. Any luck with setting batchnorm layers to eval mode. Have tried the suggested solutions but goes to NaN.</p>