Transformer decoder outputs

Arthur_Zakirov: <p>Hello everyone,</p>
<p>the goal is to use a Transformer as an autoregressive model to generate sequences.<br>
How does the decoder produce the first output prediction, if it needs the output as input in the first place?</p>
<p>That’s like “What came first, the chicken, or the egg”.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a96a289004e662dd508e44e5f19510a166f150da.png" data-download-href="https://discuss.pytorch.org/uploads/default/a96a289004e662dd508e44e5f19510a166f150da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96a289004e662dd508e44e5f19510a166f150da_2_408x500.png" alt="image" data-base62-sha1="oaI8qw7xa6DAfKNgAIpRxq6oVV8" width="408" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96a289004e662dd508e44e5f19510a166f150da_2_408x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a96a289004e662dd508e44e5f19510a166f150da_2_612x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a96a289004e662dd508e44e5f19510a166f150da.png 2x" data-dominant-color="E7E7E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">653×800 98.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My goal is to use a transformer to predict a future vehicle trajectory based on the past vehicle trajectory, not language translation, but I guess that does not change my question.<br>
My Idea would be to cut the last entry of my sequence and use it as the first decoder output. So when the x - coordinate of my vehicle is [3,4,5,6,7] I would chop off the [7] and feed it into the decoder as if it was his prediction. Then continue with regular procedure.</p>

pascal_notsawo: <p>As I mentioned in <a href="https://discuss.pytorch.org/t/inference-on-a-sequence-2-sequence-model-with-teacher-forcing/121793/3">this post</a>, a special token is usually added at the beginning of the (target) outputs during the training process, and will be used during inference to initialize the output generation (decoding) process, until another special token is produced (marking the end of this decoding process), which has also been added at the end of the target sequences during training.</p>

Arthur_Zakirov: <p>as far as I understand, the grey cell maintains the shape [batch, sequence, dim] all the way through.</p>
<p>So during infernece, how is last decoder cell supposed to give the probabilities of a single word, if the output shape is  [batch, sequence, dim] with sequence &gt; 1 ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e5d039948b8970e6b25395cb207febc82ba320a.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e5d039948b8970e6b25395cb207febc82ba320a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e5d039948b8970e6b25395cb207febc82ba320a_2_177x500.png" alt="image" data-base62-sha1="kjp578PYFUZSp49bxeDFvHuyyMO" width="177" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e5d039948b8970e6b25395cb207febc82ba320a_2_177x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e5d039948b8970e6b25395cb207febc82ba320a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e5d039948b8970e6b25395cb207febc82ba320a.png 2x" data-dominant-color="DEDDDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">233×655 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My hypothesis is the following, is that correct?</p>
<p>iteration 1: dec_inp = [start]              → dec_out = [y0]  append last out element to input (here: y0)<br>
iteration 2:  dec_inp = [start, y0]        →    dec_out = [y0, y1]      append last out element to input (here: y1)<br>
iteration 3:  dec_inp = [start, y0,y1]   →    dec_out = [y0, y1,y2]</p>
<p>So you do not actually predict one word, but the whole sequence that you have ‘so far’ and then append the last element of that sequence to the input of the new iteration.</p>
<p>Thanks in advance</p>

Arthur_Zakirov: <p>I have a question regarding your explanation in the other post:</p>
<p>" With that it will predict the first token (<code>token_1</code> ) of the output sequence, and the next thing we give to the decoder is: <code>source = encoder_output</code> and <code>target = &lt;BOS&gt; token_1</code> .<br>
It will output a second token,"</p>
<p>If input [BOS, token_1], the decoder will output also a sequence since every operation in the decoder cells maintains the shape. So how is the decoder output of the shape [batch, sequence, dim] converted into a single “second token” with the shape [batch, 1, dim], as you described? Is the sequence flattened and then the dimension adjusted with a linear layer, or do we simply take the last element of the decoder output as the new token?</p>

pascal_notsawo: <p>In fact, at the beginning of the decoding process, <code>source = encoder_output</code> and <code>target = &lt;BOS&gt;</code> are passed to the decoder and it produces <code>token_1</code>. After <code>source = encoder_output</code> and <code>target = &lt;BOS&gt; token_1</code> are still passed to the model. The problem is that the decoder will produce a representation of shape <code>[batch_size, 2, embed_dim]</code>, which we can then pass to the final classifier to have the output of shape <code>[batch_size, 2, target_vocab_size]</code>, but we just select the last word of the <code>seq_len</code> (=2 here) dimension to have a final logits of shape <code>[batch_size, target_vocab_size]</code>. We can just do an <code>argmax</code> (resp. top_k for beam search…) on it (or do <code>softmax</code> before, it doesn’t change much) to get <code>token_2</code> (resp. ), the index of the second token generated in the vocabulary.</p>
<p>You can ask me why we don’t just pass <code>source = encoder_output</code> and <code>target = token_1</code> to have outputs logits of shape <code>[batch_size, target_vocab_size]</code> directly. This is usually due to the attention mechanism (here, masked one, because the decoder) : we need to produce a representation of <code>token_1</code> taking into account all the tokens that precede it (here <code>&lt;BOS&gt;</code>), this representation is finally of shape <code>[batch_size, embedding_dim]</code>, which we can just project on the classifier to have <code>[batch_size, target_vocab_size]</code> (if in your classifier you need to look again at the representation of the previous tokens, you can pass <code>[batch_size, seq_len, embedding_dim]</code> to the classifier to have <code>[batch_size, embedding_dim, target_vocab_size]</code> and choose the last token).<br>
And this process continues until the index of the end-of-string symbol is generated</p>
<p>If you know a little bit about tensorflow, you can directly see how they illustrated it here during the validation of their model : <a href="https://www.tensorflow.org/text/tutorials/transformer#evaluate" class="inline-onebox" rel="noopener nofollow ugc">Transformer model for language understanding  |  Text  |  TensorFlow</a><br>
In pytorch, the one I use is this one, from facebook research XLM : <a href="https://github.com/facebookresearch/XLM/blob/master/xlm/model/transformer.py#L442" class="inline-onebox" rel="noopener nofollow ugc">XLM/transformer.py at master · facebookresearch/XLM · GitHub</a></p>
<p>There are many implementation of teacher forcing, beam search, … just google</p>