Can anyone help me to understand this image?

Teddy_Salas: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e85516a3f78e18c37f7c83b56ffed763004c325e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e85516a3f78e18c37f7c83b56ffed763004c325e" title="transformer_architecture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e85516a3f78e18c37f7c83b56ffed763004c325e_2_366x500.jpeg" alt="transformer_architecture" data-base62-sha1="x9j4ByaPddPwf6DCi711lYWhL9s" width="366" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e85516a3f78e18c37f7c83b56ffed763004c325e_2_366x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e85516a3f78e18c37f7c83b56ffed763004c325e_2_549x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e85516a3f78e18c37f7c83b56ffed763004c325e.jpeg 2x" data-dominant-color="E6E5E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">transformer_architecture</span><span class="informations">672×916 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Can anyone help me to understand this image ?</p>

J_Johnson: <p>Embeddings - these are learnable weights where each token(token could be a word, sentence piece, subword, character, etc) are converted into a vector, say, with 500 values between 0 and 1 that are trainable.</p>
<p>Positional Encoding - for each token, we want to inform the model where it’s located, orderwise. This is because linear layers are not ideal for handling sequential information. So we manually pass this in by adding a vector of sine and cosine values on the first 2 elements in the embedding vector.</p>
<p>This sequence of vectors goes through an attention layer, which basically is like a learnable digitized database search function with keys, queries and values. In this case, we are “searching” for the most likely next token.</p>
<p>The Feed Forward is just a basic linear layer, but is applied across each embedding in the sequence separately(i.e. 3 dim tensor instead of 2 dim).</p>
<p>Then the final Linear layer is where we want to get out our predicted next token in the form of a vector of probabilities, which we apply a softmax to put the values in the range of 0 to 1.</p>
<p>There are two sides because when that diagram was developed, it was being used in language translations. But generative language models for next token prediction just use the Transformer decoder and not the encoder.</p>
<p>Here is a PyTorch tutorial that might help you go through how it works.</p>
<p><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>And this is a tutorial that builds one from scratch in Pytorch:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://blog.floydhub.com/the-transformer-in-pytorch/">
  <header class="source">
      <img src="https://blog.floydhub.com/content/images/size/w256h256/2018/12/f_blue_small.png" class="site-icon" width="128" height="128">

      <a href="https://blog.floydhub.com/the-transformer-in-pytorch/" target="_blank" rel="noopener nofollow ugc" title="05:17PM - 09 October 2018">FloydHub Blog – 9 Oct 18</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://blog.floydhub.com/content/images/2018/10/tra.jpeg" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://blog.floydhub.com/the-transformer-in-pytorch/" target="_blank" rel="noopener nofollow ugc">How to code The Transformer in PyTorch</a></h3>

  <p>Learn how to code a transformer model in PyTorch with an English-to-French language translation task on the FloydHub blog.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>