Understanding Instance Normalization 2D with running mean and running var

Mypathissional: <p>Hi, recently I have been trying to convert StarGAN v1 from Pytorch to ONNX and they had an Instance normalization layer with track_running_stats=True. When I exported the model to ONXX it turned out that the exporter does not export the run mean/variance. Nevertheless, the onnx model still gives comparable results to the original model. I was thinking about why it can happen. Then I did the little experiment. I wanted to understand what is the difference between batch norm, instance norm with running mean/var and instance norm without the mean/var. So, I have initialized three layers with the same weights.</p>
<pre><code class="lang-auto">normTrue = nn.InstanceNorm2d(64, affine=True, track_running_stats=True)
normFalse = nn.InstanceNorm2d(64, affine=True)
input = torch.randn(10, 64, 128, 128)
bnorm = nn.BatchNorm2d(64, affine=True, track_running_stats=True)
w = torch.rand(64)
b = torch.rand(64)
m = torch.rand(64)
v = torch.rand(64)
with torch.no_grad():
    normTrue.weight=nn.Parameter(w)
    normTrue.running_mean=m
    normTrue.running_var=v
    normTrue.bias=nn.Parameter(b)
    bnorm.weight=nn.Parameter(w)
    bnorm.running_mean=m
    bnorm.running_var=v
    bnorm.bias=nn.Parameter(b)
    normFalse.weight=nn.Parameter(w)
    normFalse.bias=nn.Parameter(b)
</code></pre>
<p>It turned out that in the training mode the instance normalization with tracking running stats and without are acting the same.</p>
<pre><code class="lang-auto">with torch.no_grad():
    normoutTrue=normTrue(input).detach().cpu().numpy()
    normoutFalse=normFalse(input).detach().cpu().numpy()
    bnormout=bnorm(input).detach().cpu().numpy()
print(np.max(np.abs(normoutTrue-bnormout)))
print(np.max(np.abs(normoutTrue-normoutFalse)))
print(np.max(np.abs(bnormout-normoutFalse)))
</code></pre>
<p>0.05608654<br>
0.0<br>
0.05608654</p>
<p>But in the inference mode they are different, actually, instance norm with tracking running mean and variance is similiar to the batch norm</p>
<pre><code class="lang-auto">normTrue.eval()
normFalse.eval()
torch.onnx._export(normTrue,             # model being run
                           (torch.rand(10,64, 128, 128)),                   
                           "./norm.onnx") ;
bnorm.eval()
torch.onnx._export(bnorm,             # model being run
                           (torch.rand(10,64, 128, 128)),                   
                           "./bnorm.onnx") ;

with torch.no_grad():
    normoutTrue=normTrue(input).detach().cpu().numpy()
    normoutFalse = normFalse(input).detach().cpu().numpy()
    bnormout=bnorm(input).detach().cpu().numpy()
print(np.max(np.abs(normoutTrue-bnormout)))
print(np.max(np.abs(normoutTrue-normoutFalse)))
print(np.max(np.abs(bnormout-normoutFalse)))
</code></pre>
<p>9.536743e-07<br>
6.2534113<br>
6.2534113<br>
So, I have a hard time understanding what is the need for normalizing by run mean/var in the inference stage if they were not used in the training phase and I was actually interested whether ONNX exporter ignores running mean/variances on purpose.</p>
<p>Would be grateful for any clarification,<br>
Best Regards</p>

tom: <aside class="quote no-group" data-username="Mypathissional" data-post="1" data-topic="144139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mypathissional/48/46672_2.png" class="avatar"> Mypathissional:</div>
<blockquote>
<p>l. I was thinking about why it can happen. Then I did the little experiment. I wanted to understand what is the difference between batch norm, ins</p>
</blockquote>
</aside>
<p>Instance norm is implemented using batch norm with a reshaped input (with the batch dimension folded into the channel dimension and an artificial size 1 batch dimension being prepended) and then the running stats (which would be of shaped batch * channel) are averaged over the “batch” part.<br>
Once you accept this and use running stats, you can set the batch norm to train or not…</p>
<p>If your instance norm statistics are reasonably stable, the statistics used by the two possible modes “actual input stats” (if running stats is false or the mode is training and “collected running stats” (with running stats in eval mode) should be similar. The latter might be faster in inference because no statistics need to be gathered.</p>
<p>That said, the <code>track_running_stats=True</code> is rather not what people expect from instance norm, so I would recommend to not use it unless you have a specific reason to do so.</p>
<p>Best regards</p>
<p>Thomas</p>

Mypathissional: <p><a class="mention" href="/u/tom">@tom</a> Thanks for reaching out. Do I understand correctly that training with track_running_stats=True or False is the same, it is only different in the inference stage? Because if so then it will make sense that the outputs during the inference would be of worse quality than in the training mode.</p>
<p>Those are the results from the inference in the training/evalution mode</p>
<pre><code class="lang-auto">
def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

model= G.main
with torch.no_grad():
    outTrain = model(realInput)
    outTrain=to_numpy(outTrain)[0]
model.eval()
with torch.no_grad():
    outEval = model(realInput)
    outEval=to_numpy(outEval)[0]
    # Postprocess
    outEval=(outEval+1)/2
    outTrain =(outTrain+1)/2
plt.imshow(outTrain.transpose((1,2,0)))
plt.imshow(outEval.transpose((1,2,0)))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0_2_689x337.png" alt="image" data-base62-sha1="8ldaSE7Yft5olOXp6XSqrSBJJy8" width="689" height="337" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0_2_689x337.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0_2_1033x505.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0.png 2x" data-dominant-color="625C53"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1371×670 210 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then I guess it makes sense that ONNX exporter does not export runmean/runvariance.</p>

tom: <aside class="quote no-group" data-username="Mypathissional" data-post="3" data-topic="144139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mypathissional/48/46672_2.png" class="avatar"> Mypathissional:</div>
<blockquote>
<p><a class="mention" href="/u/tom">@tom</a> Thanks for reaching out. Do I understand correctly that training with track_running_stats=True or False is the same, it is only different in the inference stage? Because if so then it will make sense that the outputs during the inference would be of worse quality than in the training mode.</p>
</blockquote>
</aside>
<p>Yes, this is exactly my understanding as well.</p>
<blockquote>
<p>Then I guess it makes sense that ONNX exporter does not export runmean/runvariance.</p>
</blockquote>
<p>indeed.</p>
<p>Best regards</p>
<p>Thomas</p>

favorxin: <p>Hello! Do you solve the problem that the results of pytorch and onnx are different? I use the InstanceNorm3d and set the track_running_stats=False, but the onnx result and pytorch result are different when the results appear two decimal places. I am looking forward to your replay! Thank you!</p>