Confusion on how InstanceNorm1d is operating

seyeeet: <p>As long as I understand InstanceNorm1d normalizes the features for each channel.<br>
I am not sure why InstanceNorm1d works in both of the following examples and does not give me an error…</p>
<pre><code class="lang-auto">m = nn.InstanceNorm1d(5)
batch, sentence_length, embedding_dim = 2, 3, 5
embedding1 = torch.randn(batch, sentence_length, embedding_dim)
embedding2 = torch.randn(batch, embedding_dim, sentence_length)
</code></pre>
<pre><code class="lang-auto">case1) 
out1 = m(embedding1)
</code></pre>
<pre><code class="lang-auto">case2)
out2 = m(embedding2)
</code></pre>
<p>what is the difference of case 1 and 2 and and how the InstanceNorm1d is computed for each case?</p>

tom: <p>If I’m understanding your question right, you are asking why the number you pass into InstanceNorm plays no role?<br>
This is because it is only used when you</p>
<ul>
<li>either collect statistics or</li>
<li>have weight and/or bias ,</li>
</ul>
<p>neither of which instance norm does by default.</p>
<p>Best regards</p>
<p>Thomas</p>

seyeeet: <p>I am not sure, can you provide an example if possible please?</p>
<p>my question is what is happening for <code>out1 = m(embedding1)</code> and <code>out1 = m(embedding1)</code>.<br>
I dont get any error in case <code>out1 = m(embedding1)</code>, although it does not have the format that suggested by <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm1d.html" rel="noopener nofollow ugc">pytorch doc.</a><br>
I think in my example the InstanceNorm  is actually doing something.<br>
if I print <code>(out1==embedding1).all(),(out2==embedding2).all()</code> the results is <code>false, false</code>.<br>
But I am not sure what is happening actually.</p>
<p>Also, it is not clear to me what it means when you say:</p>
<aside class="quote no-group" data-username="tom" data-post="2" data-topic="138304">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<ul>
<li>either collect statistics or</li>
<li>have weight and/or bias ,</li>
</ul>
</blockquote>
</aside>
<p>even if I use <code>nn.InstanceNorm1d(in_channels, track_running_stats=False)</code> it still works okay for both cases… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

tom: <aside class="quote no-group" data-username="seyeeet" data-post="3" data-topic="138304">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seyeeet/48/21931_2.png" class="avatar"> seyeeet:</div>
<blockquote>
<p><code>nn.InstanceNorm1d(in_channels, track_running_stats=False)</code></p>
</blockquote>
</aside>
<p>It’s when you pass the (non-default) <code>track_running_stats=True</code> that you get an error.</p>
<p>The normalization should be over the last dimension, so<br>
<code>m(embedding1).mean(-1), m(embedding1).std(-1, unbiased=False)</code> gives values very close to 0 and 1.</p>
<p>Best regards</p>
<p>Thomas</p>

seyeeet: <p>I see, yes that is correct, thanks for the clarification (I think you meant <code>m(embedding2).mean(-1), m(embedding2).std(-1, unbiased=False) </code> though).<br>
so do you know when we <code>track_running_stats=False</code> what is computed in the <code>m(embedding1)</code>?</p>

tom: <p>It is precisely <code>(input - input.mean(-1, keepdim=True)) / input.std(-1, keepdim=True, unbiased=False)</code>?</p>

seyeeet: <p>Yes, now it is clear to me!<br>
Thanks a lot, Tom!</p>