Meaning of outputs (shifted right)

username: vainaijr
cooked: <p>What is the meaning of outputs (shifted right) in this image?<br>
And how would they be feeded to nn.Transformer?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4df1eb388ca66eab64efe35a269c1bbac46f7ca9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4df1eb388ca66eab64efe35a269c1bbac46f7ca9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4df1eb388ca66eab64efe35a269c1bbac46f7ca9_2_366x500.jpeg" alt="image" data-base62-sha1="b7x4AbO7PYHz8fwjL9wkts8ZVe9" width="366" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4df1eb388ca66eab64efe35a269c1bbac46f7ca9_2_366x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4df1eb388ca66eab64efe35a269c1bbac46f7ca9_2_549x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4df1eb388ca66eab64efe35a269c1bbac46f7ca9.jpeg 2x" data-dominant-color="E6E5E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">672×916 91.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: dalalaa
cooked: <p>For example, in a chatbot project, the inputs may be: “what do you want to eat” and the  outputs may be  “I want to eat apple”.</p>
<p>Then outputs(shifted right) is “BOS I want to eat apple”, and we expect the transformer to predict “I want to eat apple EOS”.<br>
BOS means begin of sentence, EOS means end of sentence.</p>

username: vainaijr
cooked: <p>what would be the way to feed input and target to nn.Transformer, and then get loss,<br>
for example, if we do,</p>
<pre><code class="lang-auto">x = nn.Transformer(10, 2) # 10 size embedding, 2 heads in nn.MultiheadAttention
x(input, target)
</code></pre>
<p>suppose we do language translation, where we know, number of words in our vocabulary in both languages.</p>
<pre><code class="lang-auto">input -&gt; 'I ate an apple'
</code></pre>
<p>and output in Kyrgyz language is</p>
<pre><code class="lang-auto">output -&gt; 'Мен алма жедим'
</code></pre>
<p>then input would be something like,</p>
<pre><code class="lang-auto">input_embedding = nn.Embedding(number of words in English language in our vocabulary, 10)
input = input_embedding(torch.LongTensor([index of words ('I', 'ate', 'an', 'apple')])
input.shape -&gt; torch.Size([4, 1, 10]) # 10 size embedding for four words
</code></pre>
<p>and target would be something like,</p>
<pre><code class="lang-auto">target_embedding = nn.Embedding(number of words in Kyrgyz language in our vocabulary, 10)
target = target_embedding(torch.LongTensor([????]) # what index do we specify here, or could we give it randomly, or do we specify index of words in correct translation itself
target.shape -&gt; torch.Size([3, 1, 10])
</code></pre>
<p>and after</p>
<pre><code class="lang-auto">z = x(input, target)
</code></pre>
<p>we get tensor of shape,</p>
<pre><code class="lang-auto">z -&gt; torch.Size([3, 1, 10])
</code></pre>
<p>which represents the 3 words, now we pass this through a linear layer.</p>
<pre><code class="lang-auto">l = nn.Linear(10, number of words in Kyrgyz language)
l(z) -&gt; torch.Size([3, 1, number of words in Kyrgyz language])
</code></pre>
<p>then pass through softmax, to get probability, or we could directly pass through crossentropyloss, and the target for this cross entropy loss, would be the index of correct translation words, like.</p>
<pre><code class="lang-auto">loss_fn = nn.CrossEntropyLoss()
loss_fn(x(input, target), target)
</code></pre>