DataLoader returns error when iteration starts

username: venkat.r
cooked: <p>I am training a BERT base model on the  <code>imdb</code>  <a href="https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews" rel="nofollow noopener">dataset</a>. However i am unable to iterate throught the Pytorch Dataloader.</p>
<p>Here is the code fully reproducible. <code>training_file</code> is the above mentioned csv.</p>
<pre><code class="lang-python">import transformers
from sklearn import model_selection
import torch
import pandas as pd
    
    
tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-cased", do_lower_case=True)
max_len = 512
train_batch_size = 8
    
"""
This class takes reviews and targets as arguments 
- Split the reviews and tokenizes
"""
class BERTDataset:
    def __init__(self, review, target):
        self.review = review
        self.target = target
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.review)

    def __getitem__(self, item):
        review = str(self.review[item])
        review = " ".join(review.split())

        tokenized_inputs = self.tokenizer.encode_plus(
            review,
            None,
            add_special_tokens=True,
            max_length=self.max_len,
            padding=True,
            truncation=True
        )

        ids = tokenized_inputs["input_ids"]
        mask = tokenized_inputs["attention_mask"]
        token_type_ids = tokenized_inputs["token_type_ids"]

        return {
            "ids": torch.tensor(ids, dtype=torch.long),
            "mask": torch.tensor(mask, dtype=torch.long),
            "token_type_ids": torch.tensor(token_type_ids, dtype=torch.long),
            "targets": torch.tensor(self.target[item], dtype=torch.float),
        }


dfx = pd.read_csv(training_file).fillna("none")
dfx['sentiment'] = dfx['sentiment'].apply(lambda x: 1 if x == 'positive' else 0)

df_train, df_valid = model_selection.train_test_split(
                        dfx,
                        test_size=0.1,
                        random_state=42,
                        stratify=dfx['sentiment'].values
                        )

# reset indices 
df_train = df_train.reset_index(drop=True)

# get ids, tokens, masks and targets  
train_dataset = BERTDataset(review=df_train['review'], target=df_train['sentiment'])

# load into pytorch dataset object
# DataLoader inputs tensor dataset of Inputs and targets 
train_data_loader = torch.utils.data.DataLoader(train_dataset, batch_size=train_batch_size, num_workers=0)

# Iterating to the Data loader
train_iter = iter(train_data_loader)
print(type(train_iter))

review, labels = train_iter.next()
</code></pre>
<p>Wehn iterating through the <code>dataloader</code> the following error comes up. Is it something to do with my dataformatting. Appreciate your inputs.</p>
<pre><code class="lang-python"> RuntimeError Traceback (most recent call last)
    &lt;ipython-input-19-c99d0829d5d9&gt; in &lt;module&gt;()
          2 print(type(train_iter))
          3 
    ----&gt; 4 images, labels = train_iter.next()
    
    5 frames
    /usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
         53             storage = elem.storage()._new_shared(numel)
         54             out = elem.new(storage)
    ---&gt; 55         return torch.stack(batch, 0, out=out)
         56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
         57             and elem_type.__name__ != 'string_':
    
    RuntimeError: stack expects each tensor to be equal size, but got [486] at entry 0 and [211] at entry 1
</code></pre>

username: mbednarski
cooked: <p>Try to set</p>
<pre><code class="lang-auto">padding='max_length'
</code></pre>
<p>This should force tokenizer to pad all sequences to <code>self.max_len</code>. Errors like <code>stack expects each tensor</code> during data loading usually means that you try to build batch from tensors with different sizes (jagged tensors are not allowed). You have either to</p>
<ol>
<li>ensure all tensors returned by <code>__getitem__</code> have same size (I used this one)</li>
<li>write custom <code>collate_fn</code> for dataloader and pad them by yourself</li>
</ol>

username: venkat.r
cooked: <p>Hi <a class="mention" href="/u/mbednarski">@mbednarski</a>,</p>
<p>Thank you so much for the fix and the explanation. This works now.</p>