Why does the transformer tutorial have a multiplication by square root of the number of inputs?

username: Brando_Miranda
cooked: <p>Why does the transformer tutorial have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there one with the output of the encoder?</p>
<p>In particular:</p>
<pre><code class="lang-auto">src = self.encoder(src) * math.sqrt(self.ninp)
</code></pre>
<p>from: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>Reference code:</p>
<pre><code class="lang-auto">import math

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import TransformerEncoder, TransformerEncoderLayer

class TransformerModel(nn.Module):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):
        super(TransformerModel, self).__init__()
        self.model_type = 'Transformer'
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = nn.Linear(ninp, ntoken)

        self.init_weights()

    def generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src, src_mask):
        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src, src_mask)
        output = self.decoder(output)
        return output
</code></pre>
<p>or</p>
<pre><code class="lang-auto">self.embedding(tokens.long()) * math.sqrt(self.emb_size)`
</code></pre>
<aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/534618/why-are-the-embeddings-of-tokens-multiplied-by-sqrt-d-note-not-divided-by-sq">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/534618/why-are-the-embeddings-of-tokens-multiplied-by-sqrt-d-note-not-divided-by-sq" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/28986/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/534618/why-are-the-embeddings-of-tokens-multiplied-by-sqrt-d-note-not-divided-by-sq" target="_blank" rel="noopener nofollow ugc">Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-networks, natural-language, transformers</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/28986/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stats.stackexchange.com/questions/534618/why-are-the-embeddings-of-tokens-multiplied-by-sqrt-d-note-not-divided-by-sq" target="_blank" rel="noopener nofollow ugc">10:27PM - 14 Jul 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/">
  <header class="source">
      <img src="https://www.redditstatic.com/desktop2x/img/favicon/android-icon-192x192.png" class="site-icon" width="192" height="192">

      <a href="https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/" target="_blank" rel="noopener nofollow ugc">reddit</a>
  </header>

  <article class="onebox-body">
    <img src="https://external-preview.redd.it/GK4J2jdDCd68cmAzq4b5v8wOyVmJzcMn471wSxWrwMY.jpg?auto=webp&amp;s=e25be8cb5cf2448d39a7e5ffc877e4d466b776d3" class="thumbnail" width="" height="">

<h3><a href="https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/" target="_blank" rel="noopener nofollow ugc">r/learnmachinelearning - Why does the embedding of tokens to the transformer...</a></h3>

  <p>2 votes and 0 comments so far on Reddit</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: AlphaBetaGamma96
cooked: <p>I’m not an expert in NLP (so double check with an expert), but I believe the square root is to normalize the encoding with respect to the input size so the embedding isn’t dependent on the size of the input. As you’re taking a dot product, the magnitude of the dot product with scale with the square-root of the dimension size and by dividing by that value you normalize it!</p>

username: Brando_Miranda
cooked: <p>hmmm…my understanding is that there are two of these going on.</p>
<ol>
<li>One inside the MHA:</li>
</ol>
<pre><code class="lang-auto">embed  = sf(QK/d**0.5)V
</code></pre>
<p>but the one here</p>
<ol start="2">
<li>is the following:</li>
</ol>
<pre><code class="lang-auto">emebd = encoder(x) * T_x**0.5
</code></pre>
<p>which seems different. If you argument holds if anything I would have expected to divide by <code>T_x**0.5</code> not multiply it… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  though your comment gave me something to think about…</p>

username: AlphaBetaGamma96
cooked: <p>I’m not too sure, what exactly is <code>T_x</code>? If I had to guess I’d say maybe you normalize out the dimension within the self-attention mechanism but add it back in after the calculation is done to preserve the dimensionality of the input?</p>

username: Brando_Miranda
cooked: <p><code>T_x = sequence length for input</code>. Usually the sequence length is denoted with a <code>t</code> e.g. <code>x^&lt;t&gt;</code> and the subscript x to emphasizes its the input.</p>

username: Brando_Miranda
cooked: <p>On a related note it seems that the other tutorial <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 1.9.0+cu102 documentation</a> also has something like that (but perhaps for a different reason):</p>
<pre><code class="lang-auto"># helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, emb_size):
        super(TokenEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, emb_size)
        self.emb_size = emb_size

    def forward(self, tokens: Tensor):
        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)
</code></pre>

username: Brando_Miranda
cooked: <p><a class="mention" href="/u/alband">@albanD</a>  apologies for the direct tag. Do you know who to tag that might know the answer to this?</p>
<hr>
<p>edit2:<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> apologies for the direct tag. Do you know who to tag that might know the answer to this?</p>

username: 111550
cooked: <p>You might don’t need this anymore, but the answer is:<br>
<a href="https://datascience.stackexchange.com/questions/87906/transformer-model-why-are-word-embeddings-scaled-before-adding-positional-encod/87909#87909" rel="noopener nofollow ugc">https://datascience.stackexchange.com/questions/87906/transformer-model-why-are-word-embeddings-scaled-before-adding-positional-encod/87909#87909</a></p>