NaN when I use batch normalization (BatchNorm1d)

davidenitti: <p>I made a module that uses the following MLP module:</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self, size_layers, activation):
        super(MLP, self).__init__()
        self.layers=[]
        self.layersnorm = []
        self.activation=activation
        for i in range(len(size_layers)-1):
            self.layers.append(nn.Linear(size_layers[i], size_layers[i + 1]))
            self.add_module('layers_' + str(i),self.layers[-1])

            self.layersnorm.append(nn.BatchNorm1d(size_layers[i + 1]))
            self.add_module('BatchNorm1d_' + str(i), self.layersnorm[-1])

    def forward(self, x):
        for i in range(len(self.layers)-1):
            if self.activation=='relu':
                x = F.relu(self.layersnorm[i](self.layers[i](x)))
            elif self.activation=='lrelu':
                x = F.leaky_relu(self.layersnorm[i](self.layers[i](x)))
            elif self.activation=='tanh':
                x = F.tanh(self.layersnorm[i](self.layers[i](x)))
        x = self.layersnorm[-1](self.layers[-1](x))
        return x
    def l1reg(self):
        w=0.
        for i in range(len(self.layers)):
            w = w + torch.sum((self.layers[i].weight).abs())
        return w
</code></pre>
<p>everything works fine without batch normalization.<br>
With batch normalization the training seem to work, but the evaluation (using model.eval()) produces NaN.<br>
is there something I’m doing wrong with batch normalization?</p>
<p>thanks!</p>

apaszke: <p>I can’t see anything obviously wrong with the model, are you sure the test data doesn’t have NaNs inside?</p>
<p>Also, it might be easier for you to use <code>nn.ModuleList</code> instead of adding the modules and maintaining a list separately.</p>

fmassa: <p>Could you provide a minimal script that reproduces the problem?</p>
<p>I can imagine having NaN during training mode if all the elements of the batch are zero, and so the mean and the std over the batch would be zero as well, leading to NaN.</p>

davidenitti: <p>thanks for the reply. the MLP code works independently but not inside another module. I’ll check what is wrong.<br>
in any case without “model.eval()” it works fine. This is strange.<br>
where do I find an example or documentation of nn.ModuleList?<br>
thank you!</p>

apaszke: <p>The docs aren’t there yet, I’ll be writing them today in the afternoon. You can construct it giving it a list of modules and that should work.</p>

Seungyoung_Park: <p>Do you find the reason why the loss becomes nan in the test mode ?<br>
I have also a similar problem.</p>

smb: <p>I also have that problem. I wanted to suggest increasing eps, which temporarily seemed to fixed the issue, but it didn’t. Is there any suggestion how to debug this? My input is fine (no nan’s).</p>

Jordan_Campbell: <p>Hello all,</p>
<p>I ran into a similar problem - I am using BatchNorm1d with a batch size of 1, which always results in <code>running_var</code>s which are NaN’s. Specifically, this only occurs with a batch of size 1.</p>
<p>This problem doesn’t occur with BatchNorm2d.</p>
<p>I thought it was possibly due to the eps value as someone suggested above, but this wouldn’t explain why it’s ok for 2d cases and why it doesn’t produce NaN’s for the first stddev calculation.</p>
<p>EDIT:</p>
<ul>
<li>I presume the NaN isn’t a result of performing <code>1 / (0 + eps)</code>? Where the 0 arises because it is computing the variance from a single example.</li>
</ul>
<p>For example:</p>
<pre><code>input = torch.FloatTensor(1,4).normal_(0,1)
bn = nn.BatchNorm1d(4)

output = bn(Variable(input))

print("output ...\n", output)
print("running mean ...\n",bn.running_mean)
print("running var ...\n",bn.running_var)
</code></pre>
<p>produces:</p>
<blockquote>
<p>output …<br>
Variable containing:<br>
0  0  0  0<br>
[torch.FloatTensor of size 1x4]</p>
</blockquote>
<blockquote>
<p>running mean …</p>
<p>0.0437<br>
0.0830<br>
0.0557<br>
0.1216<br>
[torch.FloatTensor of size 4]</p>
</blockquote>
<blockquote>
<p>running var …</p>
<p>nan<br>
nan<br>
nan<br>
nan<br>
[torch.FloatTensor of size 4]</p>
</blockquote>
<p>Have I missed something obvious?</p>
<p>Cheers,<br>
Jordan</p>

apaszke: <p>A guess would be that BatchNorm uses Bessel’s correction for variance and this makes it NaN (computed variance is 0, <code>n / (n - 1) * var = 1 / 0 * 0 = NaN</code>.</p>

apaszke: <p>Yup, just <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/THCUNN/BatchNormalization.cu#L196">found it in the sources</a>.</p>

kbrown42: <p>So if I’m understanding correctly, the solution is to use Batchnorm2d?</p>

lhatsk: <p>In case of batchnorm2d and batch size = 1. Does it work for you even in eval() mode? I’m currently using batchnorm2d with batch size = 1, but I have to stay in train() mode, otherwise the accuracy drops dramatically.</p>

cold_wind: <p>hello, did you have solved  this  problem?  if  i  should  use  the  BN1d  layer</p>

Nabarun_Goswami: <p>Hi,</p>
<p>As per the batch normalization paper,</p>
<blockquote>
<p>A model employing Batch Normalization can be trained using batch gradient descent, or Stochastic Gradient Descent with a mini-batch size m &gt; 1</p>
</blockquote>
<p>This is because of the Bessel’s correction as pointed out by Adam</p>
<blockquote>
<p>A guess would be that BatchNorm uses Bessel’s correction for variance and this makes it NaN (computed variance is 0, n / (n - 1) * var = 1 / 0 * 0 = NaN.</p>
</blockquote>
<p>So if, you can afford to use batch size &gt; 1, that would solve the NaN problem for you.</p>
<p>If you are using very small batch size or non i.i.d batches, maybe you could look at Batch Renormalization (<a href="https://arxiv.org/pdf/1702.03275.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1702.03275.pdf</a>).</p>
<p>Regards<br>
Nabarun</p>

cold_wind: <p>Hey, the totel of my test data is 10000, my batchsize is 32.  but  also only  in model.eval(),the output of bn1d  is nan</p>

Nabarun_Goswami: <p>In that case there is some other problem, most probably with your data. Batchnorm by itself will not give nan for batch sizes greater than 1. Did you scale your data? If in your training you were using float in range 0-1 and in test if its int 0-65535, your network might blowup.</p>

liubola: <p>I have sloived the problem in my case.<br>
my len(train_data) = 55937 and my batchsize = 64 &gt;&gt; 1, It looks like no problem.<br>
but I have found that 55937 % 64  = <strong>1</strong>, which means the last batchsize =1,<br>
so runing_var becomes <strong>nan</strong> after 1 epoch.<br>
hope it helps you.</p>

Zichun_Zhang: <p>But I wanna ask that here, does the n is the <code>.num_batches_tracked</code> in the BatchNorm parameters?<br>
and why is that my batch num is not 1 and still get nan,<img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=6" title=":sob:" class="emoji" alt=":sob:"></p>
<pre><code class="lang-auto">pretrain_dict['featureExtract.12.num_batches_tracked']
Out[52]: tensor(8638, device='cuda:0')
</code></pre>

Johannes_Tomasoni: <p>A cause might be your features include NaN or inf values.</p>

Alva-2020: <p>It is indeed a data problem. Thank you for your reply.</p>