Not understanding target in Transformer

imricky: <p>Hi everyone, first post here so I’m sorry if I make any mistake about the forum. I was reading this article: <a href="https://towardsdatascience.com/a-detailed-guide-to-pytorchs-nn-transformer-module-c80afbc9ffb1" class="inline-onebox" rel="noopener nofollow ugc">A detailed guide to PyTorch’s nn.Transformer() module. | by Daniel Melchor | Towards Data Science</a> about transformer and as an example he trained it to predict the continue of a sequence. What I’d like to do is reverse a sequence, by giving it as an input the sequence based on char to int dict conversion or one hit encoded array. I don’t really understand why he’s passing different target to the model (transformer) and to the loss function. I tried passing the same (the reversed sequence) to both but the model doesn’t seem to learn anything. Thanks in advance.</p>

J_Johnson: <p>This tutorial might provide better clarity:</p>
<p><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>A mask is applied in the transformer, but it has nothing to do with the targets, which is where I think the confusion may be coming in. In fact, the mask is based only on the sequence length.</p>
<p>The mask just blocks the model from seeing the next tokens at each “time step”.</p>

imricky: <p>Thanks for the answer, i already looked into that tutorial provided by pytorch. Your answer dosen’t really solve my problem because i still don’t understand in the tutorial what target he’s feeding to the model (other than the mask as you say), which apparently is different from the one he’s using for the loss function.</p>

J_Johnson: <pre><code class="lang-auto">model_input = ["Thanks", "for", "the"]
model_target = ["for", "the", "answer"] #offset by 1
</code></pre>
<p>The target in the example above is the word “answer”. The target is not fed to the model. Very simple example, but I’m not sure how else I can explain it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17b0dbdd5016fa16061f366781ebee8e45a634c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/17b0dbdd5016fa16061f366781ebee8e45a634c2" title="IMG_20230207_074444"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17b0dbdd5016fa16061f366781ebee8e45a634c2_2_690x250.jpeg" alt="IMG_20230207_074444" data-base62-sha1="3nzTOaSKovMm3Lqg3mQgVFILYyK" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17b0dbdd5016fa16061f366781ebee8e45a634c2_2_690x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17b0dbdd5016fa16061f366781ebee8e45a634c2_2_1035x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17b0dbdd5016fa16061f366781ebee8e45a634c2_2_1380x500.jpeg 2x" data-dominant-color="F9F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230207_074444</span><span class="informations">2026×735 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>