DataLoader from NLP custom dataset

Wesley_Neill: <pre><code class="lang-auto">RuntimeError: stack expects each tensor to be equal size, but got [8, 57] at entry 0 and [4, 57] at entry 1
</code></pre>
<p>I’m following this official PyTorch tutorial:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html</a></p>
<p>However, I’m altering the code as I go to be a bit more “modern” and extensible as well. So I’ve also implemented a custom dataset that in combination with a <code>DataLoader</code> is giving me the error above.</p>
<p>Here’s what I think is happening:</p>
<ol>
<li>
<p>The dataset is a bunch of names and their ethnic origins. In <code>__getitem__</code> the names are one-hot encoded at the character level. A tensor with shape <code>(len(name), len(character_vocab)</code> is returned.</p>
</li>
<li>
<p>The dataloader doesn’t like that the names are variable length and freaks out if <code>batch_size &gt; 1</code></p>
</li>
</ol>
<p><strong>Question:</strong> What technique could I use to overcome this?</p>

Unity05: <p>Yeah, your assumption is right. If I’ve understood what you try to do, then you might want to check out <a href="https://pytorch.org/docs/stable/generated/torch.cat.html" rel="noopener nofollow ugc">cat</a>.</p>

shivammehta007: <p>What I understand is that the problem is that dataloader expects inputs of equal lengths.<br>
So, to fix that you need a custom <code>collate_fn</code>.<br>
What I mean is that if you will look into docs of <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">data.DataLoader </a> it mentions</p>
<blockquote>
<p>A custom <code>collate_fn</code> can be used to customize collation, e.g., padding sequential data to max length of a batch. See <a href="https://pytorch.org/docs/stable/data.html#dataloader-collate-fn" rel="noopener nofollow ugc">this section</a> on more about <code>collate_fn</code> .</p>
</blockquote>
<p>I would suggest you change the collate function, for your case this should work!</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader

## trying to mimic your data with the input in the error
temp_data = [[[0 for _ in range(57)] for _ in range(8)], [[0 for _ in range(57)] for _ in range(4)]]


def custom_padding_collate(batch):
    """
    This method takes list of data as input of varying size and returns the batch based on max length on that batch
    Args:
        batch: input to the dataloader of batch_size argument
    Returns:
        x (torch.tensor): tensor of data of shape (batch, max_length_in_batch, embedding_dim)
        x_lengths (torch.tensor): one dimensional tensor with lengths of each element in a batch
        
    """
    batch_size = len(batch)
    x_lengths = [len(t) for t in batch]
    T = max(x_lengths)
    chacater_vocab_len = len(batch[0][0])

    for index in range(batch_size):
        batch[index] = batch[index] + [[0.0] * chacater_vocab_len] * (T - len(batch[index]))
        batch[index] = torch.tensor(batch[index])

    x = torch.stack(batch)
    x_lengths = torch.tensor(x_lengths)

    return x, x_lengths


dl = DataLoader(temp_data, batch_size=2, collate_fn=custom_padding_collate)
for x, t in dl:
    print(f"Data: {x}, Lenghts: {t}")
# This would give you your data and lengths
</code></pre>
<p>I hope this solves your problem</p>

Wesley_Neill: <p>Wow, thanks for the great response. I will look closely at this and report back!</p>