Different result between Tensor.var() vs directly calculating variance

kagok: <p>I implemented BatchNorm2d. But I found that there is difference between the way of computing variance of input. Please see the below code. Only difference between MyWrongBatchNorm and MyBatchNorm is the way of computing variance in forward().</p>
<p>MyWrongBatchNorm computes variance by using tensor.var()</p>
<pre><code class="lang-auto">class MyWrongBatchNorm(nn.Module):
    def __init__(self, num_features, momentum=0.9, epsilon=1e-05):
        super(MyWrongBatchNorm, self).__init__()
        self.momentum = momentum
        self.insize = num_features
        self.epsilon = epsilon
        
        # init weight(gamma), bias(beta),running mean, var
        self.weight = nn.Parameter(torch.ones(self.insize))
        self.bias = nn.Parameter(torch.zeros(self.insize))
        self.run_mean = torch.zeros(self.insize)
        self.run_var = torch.ones(self.insize)

    def forward(self, input, mode):
        if mode == 'train':
            mean = input.mean([0,2,3])  #mean across dims 0,2,3
            mean = mean.view(1, self.insize, 1, 1)
            var = input.var([0,2,3])  #var  across dims 0,2,3
            var = var.view(1, self.insize, 1, 1)
           
            weight = self.weight.view([1, self.insize, 1, 1])
            bias = self.bias.view([1, self.insize, 1, 1])
            out = weight*(input-mean)/torch.sqrt(var+self.epsilon) + bias

        if mode == 'test':
            pass # in this question, only consider train mode

        return out
</code></pre>
<p>MyBatchNorm directly computes variance.</p>
<pre><code class="lang-auto">class MyBatchNorm(nn.Module):
    def __init__(self, num_features, momentum=0.9, epsilon=1e-05):
        super(MyBatchNorm, self).__init__()
        self.momentum = momentum
        self.insize = num_features
        self.epsilon = epsilon
        
        # init weight(gamma), bias(beta),running mean, var
        self.weight = nn.Parameter(torch.ones(self.insize))
        self.bias = nn.Parameter(torch.zeros(self.insize))
        self.run_mean = torch.zeros(self.insize)
        self.run_var = torch.ones(self.insize)

    def forward(self, input, mode):
        if mode == 'train':
            mean = input.mean(dim=(0, 2, 3)).view(1,self.insize, 1, 1)   #mean across dims 0,2,3
            var = ((input - mean) ** 2).mean(dim=(0, 2, 3)).view(1,self.insize, 1, 1)  #var  across dims 0,2,3

            weight = self.weight.view([1, self.insize, 1, 1])
            bias = self.bias.view([1, self.insize, 1, 1])
            out = weight*(input-mean)/torch.sqrt(var+self.epsilon) + bias

        if mode == 'test':
            pass # in this question, only consider train mode

        return out
</code></pre>
<p>Then run the codes below:</p>
<pre><code class="lang-auto">x = torch.randn(100,3,32,32)
answer = nn.BatchNorm2d(x.shape[1])(x)

mybatchnorm = MyBatchNorm(num_features=x.shape[1])
y1 = mybatchnorm(x, mode='train')
print('Right result: ',torch.norm(y1-answer, p=2))

mywrongbatchnorm = MyWrongBatchNorm(num_features=x.shape[1])
y2 = mywrongbatchnorm(x)
print('Wrong result: ',torch.norm(y2-answer, p=2))
</code></pre>
<p>The results are quite different.<br>
Right result: tensor(2.8985e-05, grad_fn=)<br>
Wrong result: tensor(0.0027, grad_fn=)</p>
<p>Why is this happening? I don’t think there’s anything wrong with it, so it looks very strange.</p>
<ul>
<li>I checked that the value of variance are same on MyBatchNorm and MyWrongBatchNorm.<br>
i.e, var1 = input.var([0,2,3]).view(1, self.insize, 1, 1) and  var2 = ((input - mean) ** 2).mean(dim=(0, 2, 3)).view(1,self.insize, 1, 1) have same values.</li>
</ul>

ptrblck: <p>Batchnorm layers use the biased estimator to compute the <code>stddev</code>, so use <code>var = input.var([0,2,3], unbiased=False)</code> and the error norm drops to <code>Wrong result:  tensor(3.5124e-05, grad_fn=&lt;LinalgVectorNormBackward0&gt;)</code>.</p>

kagok: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you so much! I checked that <code>unbiased=False</code> makes correct answer. i.e, <code>Wrong result: tensor(3.5124e-05, grad_fn=&lt;LinalgVectorNormBackward0&gt;)</code>.</p>
<p>However, in MyBatchNorm class, <code>var = ((input - mean) ** 2).mean(dim=(0, 2, 3)).view(1,self.insize, 1, 1)</code> is also biased estimator (because it divides by n, not n-1). So why does MyBatchNorm output the correct answer?</p>

ptrblck: <aside class="quote no-group" data-username="kagok" data-post="3" data-topic="168881">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kagok/48/55919_2.png" class="avatar"> kagok:</div>
<blockquote>
<p>MyBatchNorm class, <code>var = ((input - mean) ** 2).mean(dim=(0, 2, 3)).view(1,self.insize, 1, 1)</code> is also biased estimator</p>
</blockquote>
</aside>
<p>The built-in <code>nn.BatchNormXd</code> as well as the corrected <code>MyWrongBatchNorm</code> and your <code>MyBatchNorm</code> layers are all using the biased variance, so I’m unsure what the issue is.<br>
Or to rephrase it and to stick closer to the argument name: none of these layers uses the unbiased statistic.</p>

kagok: <p>Ohhh, I was confused for a while. It was my mistake. There is no problem in MyBatchNorm. Thank you for your kind answer!</p>