NaN when I use batch normalization (BatchNorm1d)

davidenitti: I made a module that uses the following MLP module: <SCODE>class MLP(nn.Module):
    def __init__(self, size_layers, activation):
        super(MLP, self).__init__()
        self.layers=[]
        self.layersnorm = []
        self.activation=activation
        for i in range(len(size_layers)-1):
            self.layers.append(nn.Linear(size_layers[i], size_layers[i + 1]))
            self.add_module('layers_' + str(i),self.layers[-1])

            self.layersnorm.append(nn.BatchNorm1d(size_layers[i + 1]))
            self.add_module('BatchNorm1d_' + str(i), self.layersnorm[-1])

    def forward(self, x):
        for i in range(len(self.layers)-1):
            if self.activation=='relu':
                x = F.relu(self.layersnorm[i](self.layers[i](x)))
            elif self.activation=='lrelu':
                x = F.leaky_relu(self.layersnorm[i](self.layers[i](x)))
            elif self.activation=='tanh':
                x = F.tanh(self.layersnorm[i](self.layers[i](x)))
        x = self.layersnorm[-1](self.layers[-1](x))
        return x
    def l1reg(self):
        w=0.
        for i in range(len(self.layers)):
            w = w + torch.sum((self.layers[i].weight).abs())
        return w
<ECODE> thanks!

apaszke: I can’t see anything obviously wrong with the model, are you sure the test data doesn’t have NaNs inside?

fmassa: Could you provide a minimal script that reproduces the problem? I can imagine having NaN during training mode if all the elements of the batch are zero, and so the mean and the std over the batch would be zero as well, leading to NaN.

davidenitti: 

apaszke: The docs aren’t there yet, I’ll be writing them today in the afternoon. You can construct it giving it a list of modules and that should work.

Seungyoung_Park: 

smb: I also have that problem. I wanted to suggest increasing eps, which temporarily seemed to fixed the issue, but it didn’t. Is there any suggestion how to debug this? My input is fine (no nan’s).

Jordan_Campbell: Hello all, This problem doesn’t occur with BatchNorm2d. I thought it was possibly due to the eps value as someone suggested above, but this wouldn’t explain why it’s ok for 2d cases and why it doesn’t produce NaN’s for the first stddev calculation. EDIT: I presume the NaN isn’t a result of performing 1 / (0 + eps)? Where the 0 arises because it is computing the variance from a single example. For example: <SCODE>input = torch.FloatTensor(1,4).normal_(0,1)
bn = nn.BatchNorm1d(4)

output = bn(Variable(input))

print("output ...\n", output)
print("running mean ...\n",bn.running_mean)
print("running var ...\n",bn.running_var)
<ECODE> produces: Have I missed something obvious?

apaszke: 

apaszke: 

kbrown42: So if I’m understanding correctly, the solution is to use Batchnorm2d?

lhatsk: In case of batchnorm2d and batch size = 1. Does it work for you even in eval() mode? I’m currently using batchnorm2d with batch size = 1, but I have to stay in train() mode, otherwise the accuracy drops dramatically.

cold_wind: hello, did you have solved  this  problem?  if  i  should  use  the  BN1d  layer

Nabarun_Goswami: Hi, As per the batch normalization paper, This is because of the Bessel’s correction as pointed out by Adam So if, you can afford to use batch size > 1, that would solve the NaN problem for you.

cold_wind: Hey, the totel of my test data is 10000, my batchsize is 32.  but  also only  in model.eval(),the output of bn1d  is nan

Nabarun_Goswami: In that case there is some other problem, most probably with your data. Batchnorm by itself will not give nan for batch sizes greater than 1. Did you scale your data? If in your training you were using float in range 0-1 and in test if its int 0-65535, your network might blowup.

liubola: 

Zichun_Zhang: <SCODE>pretrain_dict['featureExtract.12.num_batches_tracked']
Out[52]: tensor(8638, device='cuda:0')
<ECODE>