Requesting help with padding/packing lstm for simple classification task

Wesley_Neill: <p>I’m running into a knowledge block.</p>
<p>I want to make a simple binary classifiyer. I took a deep dive into padded/packed sequences and think I understand them pretty well. After perusing around and thinking about it, I came to the conclusion that I should be grabbing the <em>final non-padded</em> hidden state of each sequence, so that’s what I tried below:</p>
<p><strong>Classifier</strong></p>
<pre><code class="lang-auto">
class TextClassificationModel(nn.Module):

    def __init__(self, vocab_size, embed_dim, hidden_size, num_class):
        super(TextClassificationModel, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_size, batch_first=True)
        self.fc1 = nn.Linear(hidden_size, num_class)

    def forward(self, padded_seq, lengths):
        
        # embedding layer
        embedded_padded = self.embedding(padded_seq)
        packed_output = pack_padded_sequence(embedded_padded, lengths, batch_first=True)

        # lstm layer
        output, _ = self.lstm(packed_output)
        padded_output, lengths = pad_packed_sequence(output, batch_first=True)

        # get hidden state of final non-padded sequence element:
        h_n = []
        for seq, length in zip(padded_output, lengths):
            h_n.append(seq[length - 1, :])
        
        lstm_out = torch.stack(h_n)
        
        # linear layers
        out = self.fc1(lstm_out)
        return out
</code></pre>
<p>This morning, I ported my notebook over to an IDE and ran the debugger and confirmed that <code>h_n</code> is indeed the final hidden state of each sequence, not including padding.</p>
<p>So everything runs/trains without error but my loss never decreases when I use batch size &gt; 1.</p>
<p>With <code>batch_size = 8</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5000aaef149d141aca8e2121ebb81ab0da789b44.png" data-download-href="https://discuss.pytorch.org/uploads/default/5000aaef149d141aca8e2121ebb81ab0da789b44" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5000aaef149d141aca8e2121ebb81ab0da789b44_2_690x249.png" alt="image" data-base62-sha1="bpJCWNt5aGESNt7JbPMxlIKKXQg" width="690" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5000aaef149d141aca8e2121ebb81ab0da789b44_2_690x249.png, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5000aaef149d141aca8e2121ebb81ab0da789b44.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5000aaef149d141aca8e2121ebb81ab0da789b44.png 2x" data-dominant-color="EDECF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">741×268 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With `batch_size = 1:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/8327269c6739e0e036a6a4a42e4ecf266a70ba77.png" data-download-href="https://discuss.pytorch.org/uploads/default/8327269c6739e0e036a6a4a42e4ecf266a70ba77" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8327269c6739e0e036a6a4a42e4ecf266a70ba77_2_690x254.png" alt="image" data-base62-sha1="iIeqACH7kTbPPHlZSz1fuMPqgR1" width="690" height="254" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8327269c6739e0e036a6a4a42e4ecf266a70ba77_2_690x254.png, https://discuss.pytorch.org/uploads/default/original/3X/8/3/8327269c6739e0e036a6a4a42e4ecf266a70ba77.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/3/8327269c6739e0e036a6a4a42e4ecf266a70ba77.png 2x" data-dominant-color="EDEDF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">727×268 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The reason I’m thinking I’ve done something wrong is that this performance is much worse than a simple embedding layer followed by a single linear layer:</p>
<pre><code class="lang-auto">self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
self.fc = nn.Linear(embed_dim, num_class)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/349ef8706cb26fea966479faa2796b937d4eccf7.png" data-download-href="https://discuss.pytorch.org/uploads/default/349ef8706cb26fea966479faa2796b937d4eccf7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349ef8706cb26fea966479faa2796b937d4eccf7_2_690x250.png" alt="image" data-base62-sha1="7vvq6LrUNo5r12x2lq3btSI9clV" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349ef8706cb26fea966479faa2796b937d4eccf7_2_690x250.png, https://discuss.pytorch.org/uploads/default/original/3X/3/4/349ef8706cb26fea966479faa2796b937d4eccf7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/349ef8706cb26fea966479faa2796b937d4eccf7.png 2x" data-dominant-color="EDECF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">729×265 20 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h2><a name="edit-1" class="anchor" href="#edit-1"></a>Edit:</h2>
<p>I thought I’d add the dataset, collate function and training code. This isn’t getting many hits, and I don’t know if that’s because my question is uninteresting or because I did’t provide enough data. So her’es more code:</p>
<h3><a name="dataset-2" class="anchor" href="#dataset-2"></a>Dataset:</h3>
<pre><code class="lang-auto">class TextAndLabelsFromPandas(Dataset):
    def __init__(self, dataframe):
        self.dataframe = dataframe
        self.n = 0
             
    def __getitem__(self, idx):
        text = pd.Series(self.dataframe.iloc[idx, :-1], dtype="string")
        label = self.dataframe.iloc[idx, -1]
        return label, " ".join(text.tolist())
    
    def __len__(self):
        return len(self.dataframe)
    
    def __iter__(self):
        return self;
    
    def __next__(self):
        if self.n &lt; len(self):
            result = self[self.n]
            self.n += 1
            return result
        else:
            raise StopIteration
</code></pre>
<h3><a name="collate-function-3" class="anchor" href="#collate-function-3"></a>Collate Function:</h3>
<pre><code class="lang-auto">text_pipeline = lambda x: vocab(tokenizer(x))

def collate_batch(batch):
    
    label_list, text_list = [], []
    
    # Tokenize batch
    for (label, text) in batch:
        label_list.append(label)
        processed_text = torch.tensor(text_pipeline(text), dtype=torch.int64)
        text_list.append(processed_text)
        
    # get lengths of individual sequences and sort by length 
    text_list.sort(key=lambda x: len(x), reverse=True)
    lengths = [len(sequence) for sequence in text_list]

    # convert to tensor and pad
    label_tensor = torch.tensor(label_list, dtype=torch.float64)
    padded_input_tensor = pad_sequence(text_list, batch_first=True, padding_value=vocab["&lt;pad&gt;"])
    
    return label_tensor.to(device), padded_input_tensor.to(device), lengths
</code></pre>
<h3><a name="training-protocol-4" class="anchor" href="#training-protocol-4"></a>Training Protocol:</h3>
<pre><code class="lang-auto">def train_one_epoch(model, opt, criterion, lr, trainloader):
    model.to(device)
    model.train()
    
    running_tl = 0
    
    for (label, sequence, lengths) in trainloader:
        
        opt.zero_grad()
        label = label.reshape(label.size()[0], 1)
        
        output = model(sequence, lengths)
        loss = criterion(output, label)

        running_tl += loss.item()
        loss.backward()
        opt.step()
        
    return running_tl
        
def validate_one_epoch(model, opt, criterion, lr, validloader):
    
    running_vl = 0
    
    model.eval()
    with torch.no_grad():
        for (label, sequence, lengths) in validloader:
            label = label.reshape(label.shape[0], 1)
            output = model(sequence, lengths)
            loss = criterion(output, label)
            running_vl += loss.item()
            
    return running_vl
    

def train_model(model, opt, criterion, epochs, trainload, testload=None, lr=1e-3):
    
    avg_tl_per_epoch = []
    avg_vl_per_epoch = []
    
    for e in trange(epochs):
        running_tl = train_one_epoch(model, opt, criterion, lr, trainload)
        avg_tl_per_epoch.append(running_tl / len(trainload))
        if testload:
            running_vl = validate_one_epoch(model, opt, criterion, lr, validloader)
            avg_vl_per_epoch.append(running_vl / len(testload))
    
    return avg_tl_per_epoch, avg_vl_per_epoch
            
</code></pre>

Wesley_Neill: <p>I wanted to add one more thing I’ve tried. I replaced the forward part of the model with the following:</p>
<pre><code class="lang-auto">    def forward(self, padded_seq, lengths):
        
        # embedding layer
        x = self.embedding(padded_seq)
        x = pack_padded_sequence(x, lengths, batch_first=True)

        # lstm layer
        x, hidden = self.lstm(x)
        x, lengths = pad_packed_sequence(x, batch_first=True)
        
        # linear layer1
        out = self.dropout_1(x)
        out = self.fc1(out)   
        out = self.sigmoid(out)
        
        # mask outputs to those connected with the final hidden state
        return self.mask(out, lengths)
    
    def mask(self, tensor, lengths):
        idx = torch.tile(torch.arange(max(lengths)), (tensor.shape[0], 1))
        mask = idx == torch.unsqueeze(lengths - 1, axis = 1)
        return tensor[mask]
</code></pre>
<p>This resulted in the now all too familiar loss curves:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4759a2792d33014e692f8639ab4f163be93cde0.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4759a2792d33014e692f8639ab4f163be93cde0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4759a2792d33014e692f8639ab4f163be93cde0_2_690x252.png" alt="image" data-base62-sha1="nsSgwW6QpywEs1a4lRHkvLPHkVa" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4759a2792d33014e692f8639ab4f163be93cde0_2_690x252.png, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4759a2792d33014e692f8639ab4f163be93cde0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4759a2792d33014e692f8639ab4f163be93cde0.png 2x" data-dominant-color="EDEDF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">723×265 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One thing it did help with is that it let me increase the batch size without getting those crazy loss curves. So that means I’m getting closer.</p>

Wesley_Neill: <p>So in case someone else ever has this same question, this is what a month’s worth of research has led me to believe:</p>
<p>LSTM’s are prone to overfitting unless you have a lot of data. This Kaggle competition data includes ~7600 tweets, and I just don’t think that’s enough.</p>
<p>What leads me to believe this is that I could perfectly overfit 20 samples, 10 with positive and 10 with negative labels. But when training on the whole dataset, I had two options: Overfit or Underfit.</p>
<ol>
<li>Simplify the model to the extreme. With the following parameters, the model did not overfit but severely underperformed (barely better than a flip of the coin).
<ul>
<li>Embedding size: 10</li>
<li>Lstm hidden dimension: 8</li>
<li>Added learning rate decay with a factor of 0.1 and patience of 5 epochs.</li>
</ul>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6012192a41e6d524da295760bff0d3c9f4e1d13.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6012192a41e6d524da295760bff0d3c9f4e1d13" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6012192a41e6d524da295760bff0d3c9f4e1d13_2_690x250.png" alt="image" data-base62-sha1="sfCXGhIrLR7OxmTXksu8xfq8PDl" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6012192a41e6d524da295760bff0d3c9f4e1d13_2_690x250.png, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6012192a41e6d524da295760bff0d3c9f4e1d13.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6012192a41e6d524da295760bff0d3c9f4e1d13.png 2x" data-dominant-color="EDECF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">729×265 27.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="2">
<li>
<p>A slight improvement was to train a <code>nn.EmbeddingBag</code> layer as in my original post, then transfer the trained weights over to the LSTM’s <code>nn.Embedding</code> layer.</p>
</li>
<li>
<p>Adding any more complexity to the model resulted in overfitting after ~5-30 epochs depending on the amount added.</p>
</li>
</ol>
<p>What I will try next is to make the model bidirectional, but I have no idea how to mask the loss of a bidirectional LSTM. I guess that’s a different question.</p>

Jonathan2021: <p>Hi there,<br>
First of all, thanks for the code snippet. I might reuse some of it.<br>
Second of all, I know it is late for a reply, but I noticed that you don’t seem to sort the <code>label_list</code> in your <code>collate_batch</code>. Your labels won’t match your text after you sort it. Maybe that was why your model wasn’t learning.</p>