InstanceNorm vs BatchNorm num_batches_tracked

russellizadi: <p>Is there a reason why <code>num_batches_tracked</code> gets updated in BN but not in IN?</p>
<pre><code class="lang-auto">import torch
torch.__version__
# '1.13.1'
</code></pre>
<pre><code class="lang-auto"># Create a batch of 16 data points with 2 features
x = torch.randn(16, 2, 10)
</code></pre>
<p>InstanceNorm:</p>
<pre><code class="lang-auto"># Create an instance normalization layer with track_running_stats=True
norm_layer = torch.nn.InstanceNorm1d(2, track_running_stats=True, affine=False)

# Process the input data using the normalization layer
y = norm_layer(x)

# Print the running statistics of the layer
print(norm_layer.state_dict())
# OrderedDict([('running_mean', tensor([-0.0022,  0.0019])), ('running_var', tensor([1.0003, 1.0025])), ('num_batches_tracked', tensor(0))])
</code></pre>
<p>BatchNorm:</p>
<pre><code class="lang-auto"># Create an instance normalization layer with track_running_stats=True
norm_layer = torch.nn.BatchNorm1d(2, track_running_stats=True, affine=False)

# Process the input data using the normalization layer
y = norm_layer(x)

# Print the running statistics of the layer
print(norm_layer.state_dict())
# OrderedDict([('running_mean', tensor([-0.0022,  0.0019])), ('running_var', tensor([0.9972, 1.0007])), ('num_batches_tracked', tensor(1))])
</code></pre>

marksaroufim: <p><code>num_batches_tracked</code> is simply not relevant for InstanceNorm since it doesn’t need to keep track of running statistics from multiple batches and can be applied independently per instance</p>

russellizadi: <p>Thanks Mark! I have noticed that <code>running_mean</code> and <code>running_var</code> gets updated but <code>num_batches_tracked</code> does not. My question is as one can get the <code>running_mean</code> and <code>running_var</code>, why not <code>num_batches_tracked</code>? The <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm1d.html#torch.nn.InstanceNorm1d" rel="noopener nofollow ugc">documentation</a> mentions keeping running estimates. What am I missing here?</p>
<blockquote>
<p>If <code>track_running_stats</code> is set to <code>True</code> , during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.</p>
</blockquote>
<p>For example, in the following code, the <code>num_batches_tracked</code> after the first run should be 1 as other buffers are updated.</p>
<pre><code class="lang-auto">momentum = .1
eps = 1e-5
norm_layer = torch.nn.InstanceNorm1d(2, track_running_stats=True, affine=False, momentum=momentum, eps=eps)
print(norm_layer.state_dict())
norm_layer.train()

x = torch.randn(16, 2, 10)
y = norm_layer(x)
print(norm_layer.state_dict())

m = torch.mean(x, dim=(2), keepdim=True) 
s = torch.sqrt(torch.var(x, dim=(2), keepdim=True, unbiased=False)+ eps)
y_ = (x-m) / s
print(torch.allclose(y, y_))

print((1. - momentum) * 0. + momentum * torch.mean(x, dim=(0, 2))) # == running_mean
print(torch.mean(x, dim=(0, 2))) # != running_mean
</code></pre>
<pre><code class="lang-auto">OrderedDict([('running_mean', tensor([0., 0.])), ('running_var', tensor([1., 1.])), ('num_batches_tracked', tensor(0))])
OrderedDict([('running_mean', tensor([-0.0209, -0.0072])), ('running_var', tensor([1.0197, 0.9940])), ('num_batches_tracked', tensor(0))])
True
tensor([-0.0209, -0.0072])
tensor([-0.2089, -0.0724])
</code></pre>

ptrblck: <p><code>num_batches_tracked</code> is used in batchnorm layers to update the running stats with the cumulative moving average, which is used if the <code>momentum</code> argument is set to <code>None</code>.<br>
This option is not supported in instancenorm layers as these expect a valid floating point value for the <code>momentum</code> and don’t accept <code>None</code>.</p>

russellizadi: <p>Thanks Piotr - it makes sense now. Just out of curiosity, may I ask why <code>momentum=None</code> is not an option in InstanceNorm?</p>

ptrblck: <p>Unfortunately, I don’t know why it’s disallowed or if it was just never implemented.<br>
Internally <code>batch_norm</code> would be used as seen <a href="https://github.com/pytorch/pytorch/blob/31f311a816c026bbfca622d6121d6a7fab44260d/aten/src/ATen/native/Normalization.cpp#L647-L672">here</a> so I would assume a cumulative moving average usage should be possible, but also don’t know if it wouldn’t make sense.</p>