Handling the hidden state with minibatches in a RNN for language modelling

aksellk: <p>Hi,<br>
I don’t understand how to handle the hidden state when passing minibatches of sentences into my RNN.</p>
<p>In my case the input data to the model is a minibatch of N sentences with varying length. Each sentence consist of word indices representing a word in the vocabulary:</p>
<pre><code class="lang-auto">sents = [[4, 545, 23, 1], [34, 84], [23, 6, 774]]
</code></pre>
<p>The sentences in the dataset are randomly shuffled before creating minibatches.<br>
Here is how the minibatches are created:</p>
<pre><code class="lang-auto">def batches(data, batch_size):
    """ Yields batches of sentences from 'data', ordered on length. """
    random.shuffle(data)
    for i in range(0, len(data), batch_size):
        sentences = data[i:i + batch_size]
        sentences.sort(key=lambda l: len(l), reverse=True)
        yield [torch.LongTensor(s) for s in sentences]
</code></pre>
<p>The model predicts the next element in the sentence. So the input and target looks like this:</p>
<pre><code class="lang-auto">input_sentence = [1, 4, 5, 7]
target_sentence = [4, 5, 7, 9]
</code></pre>
<p>Packed sequences are used in order to handle sentences of varying length Here is how the input and target are created:</p>
<pre><code class="lang-auto">x = nn.utils.rnn.pack_sequence([s[:-1] for s in sents])
y = nn.utils.rnn.pack_sequence([s[1:] for s in sents])
</code></pre>
<p>This input x, consisting of a minibatch of sentences, is then sent through the forward pass of the model:</p>
<pre><code class="lang-auto">out = model(x)
</code></pre>
<p>The model itself:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    """ A language model RNN with GRU layer(s). """

    def __init__(self, vocab_size, embedding_dim, hidden_dim, gru_layers, dropout):
        super(Model, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.recurrent_layer = nn.GRU(input_size=embedding_dim, hidden_size=hidden_dim,  num_layers=gru_layers, dropout=dropout, bidirectional=False)
        self.fc1 = nn.Linear(hidden_dim, vocab_size)

    def forward(self, packed_sents):
        """ Takes a PackedSequence of sentences tokens that has T tokens
        belonging to vocabulary V. Outputs predicted log-probabilities
        for the token following the one that's input in a tensor shaped
        (T, |V|).
        """
        embedded_sents = nn.utils.rnn.PackedSequence(self.embedding(packed_sents.data), packed_sents.batch_sizes)
        out_packed_sequence, hidden = self.recurrent_layer(embedded_sents)
        out = self.fc1(out_packed_sequence.data)
        return F.log_softmax(out, dim=1)
</code></pre>
<p>From this example the hidden state is not explicitly passed from one forward pass into the next.</p>
<p>Question 1)<br>
I don’t understand how to pass the hidden state from one time step to the next. I want the hidden state from item 1 in the sentence to be passed into the computation of item 2.<br>
So for the sentence [1, 2, 3], how is the hidden state after estimating element 1 passed to the computation of element 2? Is this handled under the hood by pytorch when the minibatch sentences are fed into the recurrent layer? Like here:</p>
<pre><code class="lang-auto"> self.recurrent_layer(embedded_sents)
</code></pre>
<p>I have seen some examples that explicitly feed the previous hidden state into the recurrent layer in the current forward pass. For example:</p>
<pre><code class="lang-auto">out_packed_sequence, self.hidden = self.recurrent_layer(embedded_sents, self.hidden)
</code></pre>
<p>But I don’t think that would be correct in this situation because the input is a minibatch of sentences - and then I would be using the hidden state from the previous minibatch for the current minibatch. I only want to pass the hidden state from the current item to the next item within the sentence.</p>
<p>Question 2)<br>
I think the hidden state should be reset after each new sentence. Because the dependencies I want to capture is between the items in each sentence, not between different sentences.  Is there a way to reset the hidden state after each sentence when minibatches are used? Here the entire minibatch is fed into the model for each forward pass. Or do I need to use an approach like sending one and one sentence into the model for each forward pass. Then reset the hidden state after each sentence instead of using minibatches.</p>
<p>The code is adapted from this github repo:<br>
<a href="https://github.com/florijanstamenkovic/PytorchRnnLM" rel="nofollow noopener">https://github.com/florijanstamenkovic/PytorchRnnLM</a></p>
<p>Any help is appreciated.</p>

vdw: <p>Regarding (1): Yes, everything is handled under the hood of PyTorch. If <code>embedded_sents</code> in <code>self.recurrent_layer(embedded_sents, self.hidden)</code> is the batch of the complete sentences, <code>self.hidden</code> will be updated and re-used time step by time step. If for some reason you want to intervene at each time step, you can given <code>self.recurrent</code> only sequences of length 1.</p>
<p>Regarding (2): Yes, if your batches are independent, you generally reset <code>self.hidden</code> before calling <code>foward()</code> on the next batch. You can have a look for that in <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/autoencoder/textrnnae.py" rel="nofollow noopener">some code of mine</a> for an RNN-based autoencoder. Check the method <code>init_hidden(self, batch_size)</code> and where it is called.</p>
<p>I hope that helps.</p>

aksellk: <p>Thank you for your response <a class="mention" href="/u/vdw">@vdw</a>. I think that clarifies (1).</p>
<p>Regarding (2):<br>
Yes, the hidden state should be reset after each minibatch. In the first code snippet of the model, nothing is done with the hidden state between two different forward passes.</p>
<pre><code class="lang-auto">out_packed_sequence, hidden = self.recurrent_layer(embedded_sents)
</code></pre>
<p>I think this means that the hidden state is reset after each forward pass, and therefore reset after each minibatch.</p>
<p>However, I still wonder what happens with the hidden state inside one minibatch of sentences. Suppose we have the following minibatch with two sentences: [ [1, 2, 3] [4, 5, 6] ].</p>
<p>My question is:<br>
Will the hidden state of the third item (3) from sentence 1 be passed into the computation of the first element (4) of sentence two?<br>
If the answer to this is yes, can this be a problem? I ask this based on the fact that the dataset is randomly shuffled before creating minibatches. Therefore I believe the sentences inside each minibatch should not depend on each other.</p>
<p>thanks!</p>

vdw: <p>No, the sentences in a batch are processed independently and in parallel in one go. Otherwise there would be no need for the <code>batch_size</code> dimension in the shape of the hidden state.</p>

aksellk: <p>I understand. Thank you for helping me. This answers my questions.</p>