How to use layer norm after con 1d layer?

hadaev8: <p>Cant get what to pass as argument</p>

Greg-Tarr: <p>You might want to read <a href="https://discuss.pytorch.org/t/understanding-convolution-1d-output-and-input/30764" class="inline-onebox">Understanding Convolution 1D output and Input</a></p>

hadaev8: <p>I know how 1d conv work, but i cant get what to pass to layernorm.<br>
If i pass num features (like batch norm), it expects it as last dim.</p>

vainaijr: <p>I think we have 3 options</p>
<pre><code class="lang-auto">a = nn.Conv1d(3, 3, 3) # in channels 3, out channels 3, kernel size 3
x = torch.randn(1, 3, 6) # batch size 1, 3 channels, 6 length of sequence
a(x).shape
</code></pre>
<p>torch.Size([1, 3, 4])<br>
first option</p>
<pre><code class="lang-auto">b = nn.LayerNorm([4])
</code></pre>
<p>second</p>
<pre><code class="lang-auto">b = nn.LayerNorm([3, 4])
</code></pre>
<p>third</p>
<pre><code class="lang-auto">b = nn.LayerNorm([1, 3, 4])
</code></pre>
<p>and then</p>
<pre><code class="lang-auto">b(a(x))
</code></pre>
<p>in the first case, mean, variance would be like</p>
<pre><code class="lang-auto">a.mean([2]), a.var([2], unbiased=False)
</code></pre>
<p>second case</p>
<pre><code class="lang-auto">a.mean([1, 2]), a.var([1, 2], unbiased=False)
</code></pre>
<p>third case</p>
<pre><code class="lang-auto">a.mean([0, 1, 2]), a.var([0, 1, 2], unbiased=False)
</code></pre>
<p>that is</p>
<pre><code class="lang-auto">a.mean(), a.var(unbiased=False)
</code></pre>

hadaev8: <p>So what is right way?<br>
I cant know lenght couz it is different in every batch.<br>
Here they just pass feature dim.<br>
Should [fearuredim, 1] work?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/src/transformers/modeling_distilbert.py#L69" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/src/transformers/modeling_distilbert.py#L69" target="_blank" rel="nofollow noopener">huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/src/transformers/modeling_distilbert.py#L69</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="59" style="counter-reset: li-counter 58 ;">
<li>class Embeddings(nn.Module):</li>
<li>    def __init__(self, config):</li>
<li>        super(Embeddings, self).__init__()</li>
<li>        self.word_embeddings = nn.Embedding(config.vocab_size, config.dim, padding_idx=0)</li>
<li>        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.dim)</li>
<li>        if config.sinusoidal_pos_embds:</li>
<li>            create_sinusoidal_embeddings(</li>
<li>                n_pos=config.max_position_embeddings, dim=config.dim, out=self.position_embeddings.weight</li>
<li>            )</li>
<li>
</li>
<li class="selected">        self.LayerNorm = nn.LayerNorm(config.dim, eps=1e-12)</li>
<li>        self.dropout = nn.Dropout(config.dropout)</li>
<li>
</li>
<li>    def forward(self, input_ids):</li>
<li>        """</li>
<li>        Parameters</li>
<li>        ----------</li>
<li>        input_ids: torch.tensor(bs, max_seq_length)</li>
<li>            The token ids to embed.</li>
<li>
</li>
<li>        Outputs</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

vainaijr: <p>I think layer norm is generally used after nn.Embedding because we do not want to mix one word’s embedding with another word’s embedding while normalizing.</p>
<p>I think you could go with other normalizing technique like batchnorm, if you want to use layernorm after applying conv1d, then you will have to pass size of last dim, that would be<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7606d9ba51c18ac6e23b3d9a5c170c1552f57db3.png" data-download-href="https://discuss.pytorch.org/uploads/default/7606d9ba51c18ac6e23b3d9a5c170c1552f57db3" title="Screenshot%20(198)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7606d9ba51c18ac6e23b3d9a5c170c1552f57db3.png" alt="Screenshot%20(198)" data-base62-sha1="gQ719Eqhq0yki6v3q95gboDgyHh" width="690" height="85" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(198)</span><span class="informations">869×108 6.43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hadaev8: <p>I wonna compare batchnorm with another types of norm layers.<br>
Seems like i need to use groupnorm.<br>
(not sure if give same performance)</p>

vainaijr: <p>I think doing</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 6) # batch size 1, 3 channels, 6 length of sequence
a = nn.Conv1d(3, 6, 3) # in channels 3, out channels 6, kernel size 3
gn = nn.GroupNorm(1, 6)
gn(a(x))
</code></pre>
<p>tensor([[[-0.1459,  0.5860,  0.1771,  1.1413],<br>
[-0.8613,  2.7552, -1.0135,  0.8898],<br>
[-0.1119, -0.1656, -0.4536, -0.9865],<br>
[ 0.6755, -1.3193,  1.2248, -0.5849],<br>
[ 1.2789, -0.5229,  0.1345,  0.1763],<br>
[-2.1555,  0.0149, -0.2769, -0.4565]]], grad_fn=)</p>
<p>is equivalent to</p>
<pre><code class="lang-auto">ln = nn.LayerNorm([6, 4])
ln(a(x))
</code></pre>
<p>tensor([[[-0.1459,  0.5860,  0.1771,  1.1413],<br>
[-0.8613,  2.7552, -1.0135,  0.8898],<br>
[-0.1119, -0.1656, -0.4536, -0.9865],<br>
[ 0.6755, -1.3193,  1.2248, -0.5849],<br>
[ 1.2789, -0.5229,  0.1345,  0.1763],<br>
[-2.1555,  0.0149, -0.2769, -0.4565]]],<br>
grad_fn=)</p>
<p>so we could do<br>
nn.GroupNorm(1, out_channels)<br>
and we will not have to specify Lout after applying Conv1d and it would act as second case of LayerNorm specified above.</p>
<p>So, to compare batchnorm with groupnorm or 2nd case of layernorm, we would have to replace</p>
<pre><code class="lang-auto">nn.BatchNorm1d(out_channels)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">nn.GroupNorm(1, out_channels)
</code></pre>

miken: <p>Hello, <a class="mention" href="/u/vainaijr">@vainaijr</a>, I am running into the same issue. Could you possibly just transpose the tensor from (batch, feature, seq_len) —&gt; (batch, seq_len, features)? then feed that through layernorm, and then re-transpose back? Would this work?</p>

vainaijr: <p>yes I think it would work, for example,</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 4) # 3 words, each represented by four size embedding
# if we had (batch, features, seq_len), then we transpose it to (batch, seq_len, features)
x
</code></pre>
<p>tensor([[[-0.0247, -0.0365,  0.0992,  0.8617],<br>
[ 0.9550,  1.0243, -0.7017, -1.1300],<br>
[ 1.2312,  1.0802,  0.3377,  1.8007]]])</p>
<pre><code class="lang-auto">y = nn.LayerNorm([4]) 
# if we do not want to normalize one word based on other word.
# normalize based on individual word representation
y(x)
</code></pre>
<p>tensor([[[-0.6720, -0.7037, -0.3385,  1.7142],<br>
[ 0.9514,  1.0232, -0.7654, -1.2092],<br>
[ 0.2275, -0.0618, -1.4847,  1.3190]]],<br>
grad_fn=)</p>
<p>which is equivalent to result of,</p>
<pre><code class="lang-auto">((x.permute(2, 0, 1) - x.mean([2]))/(x.var([2], unbiased=False) + 1e-05).sqrt()).permute(1, 2, 0)
</code></pre>

JayceWong: <p>LN and 1 group GN are not equivalent. You can use <code>torch.dist()</code> to check the difference</p>