Pytorch Transformers

Anandh_Perumal_Konar: <p>The below code is from pytorch <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">doc</a></p>
<pre><code class="lang-auto">    def forward(self, src):
        if self.src_mask is None or self.src_mask.size(0) != len(src):
            device = src.device
            mask = self._generate_square_subsequent_mask(len(src)).to(device)
            self.src_mask = mask

        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src, self.src_mask)
        output = self.decoder(output)
        return output

</code></pre>
<p>I don’t understand why there is a mask in the transformer encoder?<br>
In decoder, we don’t want the decoder to look at the future and it makes sense to mask but why encoder?</p>
<p>and what is the use of the below code?<br>
<code>math.sqrt(self.ninp)</code> in</p>
<pre><code class="lang-auto">src = self.encoder(src) * math.sqrt(self.ninp)
</code></pre>
<p>Thanks in advance</p>

vainaijr: <p>I created one post for a similar question here,</p><aside class="quote" data-post="1" data-topic="63390">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gocurrent/48/21496_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-add-padding-mask-to-nn-transformerencoder-module/63390">How to add padding mask to nn.TransformerEncoder module?</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    I want to use vanilla transformer(only the encoder side), but I don’t know how&amp;where to add the padding mask.
  </blockquote>
</aside>

<p>see if it helps, regarding <code>math.sqrt(self.ninp)</code><br>
in the paper <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1706.03762.pdf</a>, they use a scaling factor of <code>math.sqrt(self.ninp)</code>, maybe cancelling this scaling factor (or not considering to divide by <code>math.sqrt(self.ninp)</code>), gives a better accuracy in the tutorial.</p>

Anandh_Perumal_Konar: <aside class="quote no-group" data-username="vainaijr" data-post="2" data-topic="76993">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vainaijr/48/49167_2.png" class="avatar"> vainaijr:</div>
<blockquote>
<p>not considering to divide by <code>math.sqrt(self.ninp)</code></p>
</blockquote>
</aside>
<p>Well, actually my question is different from the one you asked.<br>
My question is what’s the need for mask in the encoder layer?<br>
But I think it makes sense when you want to use an architecture like GPT 2, I guess. Correct me if I’m wrong.</p>
<p>Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.</p>
<p>But I don’t understand why are we multiplying it? it should be divided based on the attention paper.</p>

ShaneB: <aside class="quote no-group" data-username="Anandh_Perumal_Konar" data-post="3" data-topic="76993">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anandh_perumal_konar/48/12027_2.png" class="avatar"> Anandh_Perumal_Konar:</div>
<blockquote>
<p>Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.</p>
<p>But I don’t understand why are we multiplying it? it should be divided based on the attention paper.</p>
</blockquote>
</aside>
<p>Again, regarding the scaling factor:</p>
<p>The <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noopener nofollow ugc">paper</a> does say:</p>
<blockquote>
<p>In the embedding layers, we multiply those weights by <code>sqrt(d_model)</code>.</p>
</blockquote>
<p>(Note <a href="https://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noopener nofollow ugc">The Annotated Transformer</a> directly connects the paper to PyTorch code line-by-line).</p>
<p>However, the paper does not say <em>why</em> this is done.</p>
<p>This <a href="https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec" rel="noopener nofollow ugc">post</a> says the following:</p>
<blockquote>
<p>The reason we increase the embedding values [by adding <code>math.sqrt(self.d_model)</code>] before addition is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.</p>
</blockquote>