Collate_fn function to handle different text size

adama: <p>Hi,</p>
<p>I know there are a many questions about the <code>collate_fn()</code> function to make inputs the same shape but I still have problems to understand it and use it.</p>
<p>Because all my text is longer than 512 tokens I need to cut them into smaller pieces. So i applied a sliding window in the <code>__getitem__</code> function. Here is my <code>Dataset</code> class.</p>
<pre><code class="lang-auto">MAX_LEN = 400
STRIDE = 20

class CustomDataset(Dataset):

    def __init__(self, dataframe, tokenizer, max_len, stride):
        self.tokenizer = tokenizer
        self.data = dataframe
        self.text = dataframe.text
        self.targets = self.data.labels
        self.max_len = max_len
        self.stride = stride

    def __len__(self):
        return len(self.text)

    def __getitem__(self, index):
        text = str(self.text[index])
        text = " ".join(text.split())

        inputs = self.tokenizer(
            text,
            None,
            max_length=MAX_LEN,
            stride=STRIDE,
            padding='max_length',
            truncation='only_first',
            return_overflowing_tokens=True,
            return_tensors='pt'
        )
        ids = inputs['input_ids']
        mask = inputs['attention_mask']
        token_type_ids = inputs["token_type_ids"]

        return {

            'ids': torch.tensor(ids, dtype=torch.long),
            'mask': torch.tensor(mask, dtype=torch.long),
            'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),
            'targets': torch.tensor(self.targets[index], dtype=torch.float)
        }
</code></pre>
<p>As you can see I return a dict with <code>ids</code>,<code> attention_mask</code>, <code>token_type_ids</code> and the <code>target</code>.<br>
For example if I use <code>batch_size = 8</code> this could be the potential data</p>
<pre><code class="lang-auto">|ID´s: torch.Size([971, 400]) | Mask: torch.Size([971, 400]) | TokenTypeID´s: torch.Size([971, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([17792, 400]) | Mask: torch.Size([17792, 400]) | TokenTypeID´s: torch.Size([17792, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([177, 400]) | Mask: torch.Size([177, 400]) | TokenTypeID´s: torch.Size([177, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([402, 400]) | Mask: torch.Size([402, 400]) | TokenTypeID´s: torch.Size([402, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([11, 400]) | Mask: torch.Size([11, 400]) | TokenTypeID´s: torch.Size([11, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([48, 400]) | Mask: torch.Size([48, 400]) | TokenTypeID´s: torch.Size([48, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([7, 400]) | Mask: torch.Size([7, 400]) | TokenTypeID´s: torch.Size([7, 400]) | Targets: torch.Size([6])|
|ID´s: torch.Size([31, 400]) | Mask: torch.Size([31, 400]) | TokenTypeID´s: torch.Size([31, 400]) | Targets: torch.Size([6])|
</code></pre>
<p>Because the shapes are not the same I can´t load them into the network. If I understand it right, I can use the <code>collate_fn()</code> function to bring all data into the same shape.</p>
<p>I wrote a custom function as desribed <a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418/13">here</a></p>
<pre><code class="lang-auto">def pad_collate(batch):
    data = [item['ids'] for item in batch]
    data = pack_sequence(data, enforce_sorted=False)
    targets = [item['targets'] for item in batch]
    return data, targets
</code></pre>
<p>which returns something like this:</p>
<pre><code class="lang-auto">(PackedSequence(data=tensor([[    3,  2843,  6406,  ..., 10437,    59,     4],
        [    3, 10696, 26897,  ...,  1464,   248,     4],
        [    3,  3396,  7083,  ..., 26971, 26924,     4],
        ...,
        [    3,   115,  1225,  ..., 26935, 15070,     4],
        [    3,    62, 26914,  ...,    21, 16923,     4],
        [    3, 26900,   860,  ...,     0,     0,     0]]), batch_sizes=tensor([2, 2, 2,  ..., 1, 1, 1]), sorted_indices=tensor([1, 0]), unsorted_indices=tensor([1, 0])), [tensor([1., 0., 0., 0., 0., 0.]), tensor([0., 0., 0., 1., 0., 0.])])
</code></pre>
<p>I can´t understand the output, can you help me with that?<br>
Also this only pads the text to the longest sequence, but what´s about the <code>mask</code> and <code>token_type_ids</code>?<br>
How can I extend the <code>pad_collate()</code> function with <code>mask</code> and<code> token_type_ids</code> so that I can feed the data into network and train it like that:</p>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    for _, data in enumerate(training_loader, 0):
        ids = data['ids'].to(device, dtype=torch.long)
        mask = data['mask'].to(device, dtype=torch.long)
        token_type_ids = data['token_type_ids'].to(device, dtype=torch.long)
        targets = data['targets'].to(device, dtype=torch.float)

        outputs = model(ids, mask, token_type_ids)

        optimizer.zero_grad()
        loss = loss_fn(outputs, targets)
        if _ % 5000 == 0:
            print(f'Epoch: {epoch}, Loss:  {loss.item()}')

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

for epoch in range(EPOCHS):
    train(epoch)
</code></pre>