Passing a minibatchs of sequential data through a bidirectional rnn

aksellk: <p>Hi, can someone please help me by explaining how to correctly pass minibatchs of sequential data through a bidirectional rnn? And perhaps show an example, if possible?</p>
<p>I will try to provide some context for my problem:</p>
<p>The problem is similar to a language modeling task. I’m trying to predict the next item for each item in a sequence.<br>
In my case the input data to the model is a minibatch of N sentences with varying length. Each sentence consist of word indices representing a word in the vocabulary:</p>
<pre><code class="lang-auto">sents = [[4, 545, 23, 1], [34, 84], [23, 6, 774]]
</code></pre>
<p>The sentences in the dataset are randomly shuffled before creating minibatches.<br>
Here is how the minibatches are created:</p>
<pre><code class="lang-auto">def batches(data, batch_size):
    """ Yields batches of sentences from 'data', ordered on length. """
    random.shuffle(data)
    for i in range(0, len(data), batch_size):
        sentences = data[i:i + batch_size]
        sentences.sort(key=lambda l: len(l), reverse=True)
        yield [torch.LongTensor(s) for s in sentences]
</code></pre>
<p>The model predicts the next element in the sentence. So the input and target looks like this:</p>
<pre><code class="lang-auto">input_sentence = [1, 4, 5, 7]
target_sentence = [4, 5, 7, 9]
</code></pre>
<p>Packed sequences are used in order to handle sentences of varying length Here is how the input and target are created:</p>
<pre><code class="lang-auto">x = nn.utils.rnn.pack_sequence([s[:-1] for s in sents])
y = nn.utils.rnn.pack_sequence([s[1:] for s in sents])
</code></pre>
<p>This input x, consisting of a minibatch of sentences, is then sent through the forward pass of the model:</p>
<pre><code class="lang-auto">out = model(x)
</code></pre>
<p>The model itself:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    """ A language model RNN with GRU layer(s). """

    def __init__(self, vocab_size, embedding_dim, hidden_dim, gru_layers, dropout):
        super(Model, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.recurrent_layer = nn.GRU(input_size=embedding_dim, hidden_size=hidden_dim,  num_layers=gru_layers, dropout=dropout, bidirectional=True)
        self.fc1 = nn.Linear(hidden_dim*2, vocab_size)

    def forward(self, packed_sents):
        """ Takes a PackedSequence of sentences tokens that has T tokens
        belonging to vocabulary V. Outputs predicted log-probabilities
        for the token following the one that's input in a tensor shaped
        (T, |V|).
        """
        embedded_sents = nn.utils.rnn.PackedSequence(self.embedding(packed_sents.data), packed_sents.batch_sizes)
        out_packed_sequence, hidden = self.recurrent_layer(embedded_sents)
        out = self.fc1(out_packed_sequence.data)
        return F.log_softmax(out, dim=1)
</code></pre>
<p>The output of the model is a probability distribution over the unique items in the dataset, representing how likely they are to represent the next item in the sequence for each item in the sequence.</p>
<p>Around 100 000 sequences are used for training the model, and 100 000 are used for testing the model.</p>
<p>The model is evaluated using top-k accuracy/precision@K:</p>
<pre><code class="lang-auto">def topk_accuracy(output_distribution, targets, k):
    _, pred = torch.topk(input=output_distribution, k=k, dim=1)
    pred = pred.t()
    correct = pred.eq(targets.expand_as(pred))
    return correct.sum().item() / targets.shape[0]
</code></pre>
<p>I have evaluated the model by using different variations of the recurrent layer. That is GRU, LSTM, bidirectional GRU and Bidirectional LSTM.</p>
<p>The code for training the model:<br>
<a href="https://github.com/aksellk/recsys_telenor/blob/master/prediction/main.py" rel="noopener nofollow ugc">https://github.com/aksellk/recsys_telenor/blob/master/prediction/main.py</a></p>
<p>The model itself:<br>
<a href="https://github.com/aksellk/recsys_telenor/blob/master/prediction/model.py" rel="noopener nofollow ugc">https://github.com/aksellk/recsys_telenor/blob/master/prediction/model.py</a></p>
<p>The code for evaluating the model:<br>
<a href="https://github.com/aksellk/recsys_telenor/blob/master/prediction/testing.py" rel="noopener nofollow ugc">https://github.com/aksellk/recsys_telenor/blob/master/prediction/testing.py</a></p>
<p>When bidirectional GRU/LSTM is used, the top1-accuracy is around 94%<br>
When plain GRU/LSTM is used, the top1-accuracy is around 37%<br>
I suspect something in my experiments is wrong because the bidirectional model achieves too good results compared to the plain versions of GRU/LSTM.</p>
<p>I have also made an experiment where I calculate the average precision@1 for each item at specific positions in the sessions. For example, the average precision@1 for the first item in the session over all sessions (Then for the second position, the third, and so on). The idea is to investigate how the prediction of the model changes throughout the sessions.</p>
<p>The code for this experiment is in a jupyter notebook:<br>
<a href="https://github.com/aksellk/recsys_telenor/blob/master/prediction/average_precision_for_the_nth_interaction.ipynb" rel="noopener nofollow ugc">https://github.com/aksellk/recsys_telenor/blob/master/prediction/average_precision_for_the_nth_interaction.ipynb</a></p>
<p>Here is a plot from this experiment showing the top1-accuracy of the bidirectional LSTM model and the plain LSTM model. The x-axis represents the position of the items in the sequences:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/74305363d67f85fc40e824f96fc11834901fb90a.png" data-download-href="https://discuss.pytorch.org/uploads/default/74305363d67f85fc40e824f96fc11834901fb90a" title="first_n_acc_lstm-1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/74305363d67f85fc40e824f96fc11834901fb90a_2_690x459.png" alt="first_n_acc_lstm-1" data-base62-sha1="gzQVhQok7bIoOVOQGoie5gvytiq" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/74305363d67f85fc40e824f96fc11834901fb90a_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/74305363d67f85fc40e824f96fc11834901fb90a_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/74305363d67f85fc40e824f96fc11834901fb90a.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">first_n_acc_lstm-1</span><span class="informations">1200×800 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I suspect there is something wrong with the experiment because the bidirectional model shows a top1-accuracy over 90% for each item in the sequence, which I believe is too promising.<br>
Further, the top-1 accuracy for the first items in each sequence should be affected by the cold-start problem. For the plain LSTM, we see that the average top1-accuarcy for the first item is much lower compared to the rest of the items. But this is not the case for the bidirectional problem, it looks like it is too high.</p>
<p>So to sum up, I believe there is something wrong in how I pass sequences in minibatches through the bidirectional model and therefore ask for help for understanding how to do this properly. And possibly any ideas as to why the bidirectional variation of the model shows so high topK-accuarcy.</p>
<p>This question is related to a previously asked question on the forum:<br>
<a href="https://discuss.pytorch.org/t/handling-the-hidden-state-with-minibatches-in-a-rnn-for-language-modelling/44261/4">https://discuss.pytorch.org/t/handling-the-hidden-state-with-minibatches-in-a-rnn-for-language-modelling/44261/4</a></p>
<p>If something is unclear, please feel free to ask me.<br>
Any help is very much appreciated.<br>
Thanks!</p>

vdw: <p>Here’s what I think(!) is going on:</p>
<p>You want to train a network that takes a sequence of words as input to predict the next word (many-to-one). But what you’re actually training is a many-to-many network, more specifically a sequence tagging network like for POS or NER tagging.</p>
<p>Given your example, you’re data item should look more like:</p>
<pre><code>input_sentence = [1, 4, 5, 7]
target_word = 9
</code></pre>
<p>Why should the network learn that 1 maps to 4, 4 maps to 5, and so on. That’s not the task, but it affects your loss and hence what your network learns.</p>
<p>In my opinion, that would also explain why the Bi-LSTM is so much better than the LSTM. Since the Bi-LSTM starts also from the end of the sentence – given a sentence of length N – it will learn that for the last N-1 steps, the last input will be the next target. The simple LSTM would have to look into the future for that.</p>
<p>So this is what I would try:</p>
<ul>
<li>Change your dataset such that one data item is a sequence as input and a single word as output, and treat it like a classification (many-to-one) task</li>
<li>Take <code>hidden</code> and not <code>out_packed_sequence</code> as input for the <code>fc1</code> layer! Since you only want the last state for many-to-one, the last output in case of Bi-LSTM are on “opposite ends” (see also <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">my post</a>). So it’s much simpler to use <code>hidden</code>. Here’s an <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="nofollow noopener">example for an RNN-based classifier</a> that might help. In your case, <code>label_size</code> will also be <code>vocab_size</code>.</li>
</ul>
<p>I hope that helps.</p>

aksellk: <p>Thank you, <a class="mention" href="/u/vdw">@vdw</a>! I believe you are right. The data item should be represented like you said:</p>
<pre><code class="lang-auto">input_sentence = [1, 4, 5, 7]
target_word = 9
</code></pre>
<p>But I also think the prediction should be repeated for each item in the sequence. The reason why I think this is because if we only consider the last element in the sequence, it is the element with the longest sequence of previous elements. So in order to make a fair prediction, it should be repeated for each of the next items in the sequences. Splitting the sequences like this:</p>
<pre><code class="lang-auto">input_sentence = [1]
target_word = 4

input_sentence = [1, 4]
target_word = 5

input_sentence = [1, 4, 5]
target_word = 7

input_sentence = [1, 4, 5, 7]
target_word = 9
</code></pre>
<p>But I think splitting sequences like this can be hard to combine with minibatching during training. Especially since the sequences have varying lengths.</p>
<p>My idea is therefore to carry out this sequence split before I train and test the model. That is before loading the dataset. And it would lead to that the size of the dataset would increase. I believe this would make it easier to combine session splitting with minibatching.  Do you think that would make sense?</p>
<p>This is how I would create the dataset:</p>
<pre><code class="lang-auto">'''
Splitting the sequences and adds more training data like this:

[1, 2]
[1, 2, 3]
[1, 2, 3, 4]
'''
def next_item_sequence_split(data):
    new_data = []
    for d in data:
        splits = [torch.LongTensor(d[:i]) for i in range(2,len(d)+1)]
        new_data.extend(splits)
    random.shuffle(new_data)
    return new_data

'''
Splitting the data into 80% training and 20% testing
'''
def split_dataset(dataset):
    random.shuffle(dataset)
    train_size = int(0.8 * len(dataset))
    train = dataset[0:train_size]
    test = dataset[train_size:-1]
    return train, test 

# split data into training and testing, x is a list of sequences
train, test = split_dataset(x)

x_train = next_item_sequence_split(train)
x_test = next_item_sequence_split(test)
</code></pre>
<p>Thanks!</p>

vdw: <p>If the splitting makes sense depends on your data and task.</p>
<p>You can have a look at <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284/13">this thread</a> to generate batches of sequences of equal length. I use this all the time since it’s worry-free: neither padding nor packing is needed.</p>

aksellk: <p>Thank you! I will have a look at it.</p>