How to speed up the RNN in the for loop

Sine: <p>Hi, I am trying to use GRU to predict tokens. However, the computation speed is very slow in for loop.</p>
<p>My code is as below:</p>
<pre><code class="lang-auto">decoded_words, word_distributions = [], []

for _ in range(max_utter_len):
    word_distribution, decoder_h = self.decoder(
        decoder_inputs, contexts, decoder_h)

    word_distribution = word_distribution.squeeze(1)
    # [batch size, voc size]

    top1 = word_distribution.topk(1)[1]
    # [batch size, 1]

    decoded_words.append(top1.squeeze(-1))
    word_distributions.append(word_distribution)

    decoder_inputs = top1

decoded_words = torch.stack(decoded_words, dim=-1)
word_distributions = torch.stack(word_distributions, dim=1)
</code></pre>
<p>It takes 17 seconds for every 100 iterations.</p>
<p>However, I use the “teacher forcing” in the training, and it only takes 7 seconds for every 100 iterations.<br>
The code is as below:</p>
<pre><code class="lang-auto">word_distributions, decoder_init_hidden = self.decoder(
            inputs, context, decoder_init_hidden)
</code></pre>
<p>How to speed up RNN prediction?<br>
Thank you.</p>

Abhilash_Srivastava: <p>I am guessing this is only for the inference part. The training speed for both the first case and the second case is still pretty similar?</p>
<p>Remember teacher forcing can be used only during training and not during inference.</p>
<p>What you are doing in the first case for inference looks fine to me. If you don’t want to break on <code>&lt;EOS&gt;</code> token, you can prepare a list of the required size (<code>max_utter_len</code>) and then fill it using <code>decoded_words[i] = top1.squeeze(-1)</code> and <code>word_distributions[i] = word_distribution</code>. This will give marginal improvement.</p>

Sine: <aside class="quote no-group" data-username="Abhilash_Srivastava" data-post="2" data-topic="106788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhilash_srivastava/48/33127_2.png" class="avatar"> Abhilash_Srivastava:</div>
<blockquote>
<p>I am guessing this is only for the inference part. The training speed for both the first case and the second case is still pretty similar?</p>
</blockquote>
</aside>
<p>They differ by 10 seconds.</p>
<aside class="quote no-group" data-username="Abhilash_Srivastava" data-post="2" data-topic="106788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhilash_srivastava/48/33127_2.png" class="avatar"> Abhilash_Srivastava:</div>
<blockquote>
<p>Remember teacher forcing can be used only during training and not during inference.</p>
</blockquote>
</aside>
<p>I know it. I only use teacher forcing in my training.</p>
<aside class="quote no-group" data-username="Abhilash_Srivastava" data-post="2" data-topic="106788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhilash_srivastava/48/33127_2.png" class="avatar"> Abhilash_Srivastava:</div>
<blockquote>
<p>What you are doing in the first case for inference looks fine to me. If you don’t want to break on <code>&lt;EOS&gt;</code> token, you can prepare a list of the required size (<code>max_utter_len</code>) and then fill it using <code>decoded_words[i] = top1.squeeze(-1)</code> and <code>word_distributions[i] = word_distribution</code>. This will give marginal improvement.</p>
</blockquote>
</aside>
<p>Appreciate it. I will try it.</p>