Inference on a Sequence-2-Sequence model with teacher forcing

krishnab: <p>I was watching some very good videos by <a href="https://www.youtube.com/watch?v=EoGUlvhRYpk&amp;t=1285s" rel="noopener nofollow ugc">Aladdin Persson</a> on Youtube, and he shows a simple Sequence-2-Sequence model for machine translation + Teacher Forcing. Now technically I adapted this model for time-series analysis, but the example is fine. The original code is below. The key issues is that due to Teacher Forcing, in the <code>Seq2Seq</code> layer, the <code>forward()</code> method takes both the input sentence and the label–meaning the correct answer.</p>
<p>My question is, in the case of actual inference on the model, I won’t have a label. During inference I will only have the input sentence. So when trying to run the model, the model function will expect <code>model(input, label)</code>, and we won’t have any label to provide. So what is the way to deal with that?</p>
<p>Here is the code.</p>
<pre><code class="lang-auto">class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder):
        super(Seq2Seq, self).__init__()
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, source, target, teacher_force_ratio=0.5):
        batch_size = source.shape[1]
        target_len = target.shape[0]
        target_vocab_size = len(english.vocab)

        outputs = torch.zeros(target_len, batch_size, target_vocab_size).to(device)

        hidden, cell = self.encoder(source)

        # Grab the first input to the Decoder which will be &lt;SOS&gt; token
        x = target[0]

        for t in range(1, target_len):
            # Use previous hidden, cell as context from encoder at start
            output, hidden, cell = self.decoder(x, hidden, cell)

            # Store next output prediction
            outputs[t] = output

            # Get the best word the Decoder predicted (index in the vocabulary)
            best_guess = output.argmax(1)

            # With probability of teacher_force_ratio we take the actual next word
            # otherwise we take the word that the Decoder predicted it to be.
            # Teacher Forcing is used so that the model gets used to seeing
            # similar inputs at training and testing time, if teacher forcing is 1
            # then inputs at test time might be completely different than what the
            # network is used to. This was a long comment.
            x = target[t] if random.random() &lt; teacher_force_ratio else best_guess

        return outputs
</code></pre>
<p>As you can see, the <code>forward()</code> function takes a <code>source, target</code>, where the source is the input sentence and the target is the actually translated sentence. I have to use the model as below.</p>
<pre><code class="lang-auto">model = Seq2Seq(encoder_net, decoder_net).to(device)
prediction = model(data, label)
</code></pre>
<p>Can anyone explain how to do inference on a Sequence-to-Sequence model, or if there is a better way to train or write these models to deal with teacher forcing, etc. Thanks.</p>

Kushaj: <p>During inference, you use the encoder as normal. For decoder, you pass the input (output of encoder) and then the output of decoder is used as input for next timestamp (and repeat).</p>
<p>So in your seq2seq imagine an arrow going from decoder output at step <code>t</code> to decoder input at step <code>t+1</code>.</p>

pascal_notsawo: <p>Adding to what <a class="mention" href="/u/kushaj">@Kushaj</a>  says, special symbols are usually added to the sequence when forming them, in most cases: <code>&lt;BOS&gt; target_senquence &lt;EOS&gt;</code></p>
<p>Now in production, once we have the encoder output for the input sequence so we want to predict the output (<code>encoder_output</code>), the first thing we give to the decoder is: <code>source = encoder_output</code> and <code>target = &lt;BOS&gt;</code></p>
<p>With that it will predict the first token (<code>token_1</code>) of the output sequence, and the next thing we give to the decoder is: <code>source = encoder_output</code> and <code>target = &lt;BOS&gt; token_1</code>.<br>
It will output a second token, and so on until it outputs <code>&lt;EOS&gt;</code>, and the decoding process stops: the sequence between <code>&lt;BOS&gt;</code> and <code>&lt;EOS&gt;</code> will be our prediction.</p>
<p>In practice, to predict each token generally, we don’t take only one token that maximizes the probability at the output of the model, but k (<code>top_k</code>) tokens that maximize it: thus several possible outputs are exploited before choosing the one with the maximum probability.<br>
The idea of beam search here is that, it is not necessarily because the current token produced by the model has maximized the output probability, that it will allow to have the final output sequence that maximizes the probability: several paths are thus exploited to hope to be able to make the good choice.</p>