Concat dataset and preserving their index

username: kwcckw
cooked: <p>Hi,</p>
<p>I’m trying to concatenate more than 1 dataset and after the concatenation, looks like the index of the dataset would be not in order.</p>
<p>I’m having 6 images and labels related datasets, and want to pass them in a similar sequence.</p>
<p>This is the length of each dataset:<br>
666<br>
80<br>
59<br>
39<br>
22<br>
72</p>
<p>and this is how i concatenate them:<br>
train_all = torch.utils.data.ConcatDataset({train_good,train_blow_hole,train_break,train_crack,train_fray,train_uneven})</p>
<p>train_all_label = torch.utils.data.ConcatDataset({train_label_good,train_label_blow_hole,train_label_break,train_label_crack,train_label_fray,train_label_uneven})</p>
<p>After the concatenation:</p>
<p>print(train_all.cumulative_sizes)<br>
[72, 738, 797, 877, 899, 938]</p>
<p>print(train_all_label.cumulative_sizes)<br>
[72, 111, 777, 836, 858, 938]</p>
<p>The index is not the same anymore and this would causing issue in the training stage later. Is there an efficient way to concatenate the dataset while preserving their index in the concatenated dataset?</p>
<p>Thanks</p>

username: kwcckw
cooked: <p>is there anyone have any idea to solve the issue or possibly to provide some reference?</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>I am not super familiar with <code>ConcatDataset</code>, but after looking at the <a href="https://pytorch.org/docs/1.3.0/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">docs</a>, here is how I would proceed in your situation. First, I would create a <code>CustomDataset</code> class that inherits PyTorch’s <code>torch.utils.data.Dataset</code> class so that <code>CustomDataset</code> has both the data and the target of each individual dataset. Second, once you have your 6 <code>CustomDataset</code> objects, you can use <code>torch.utils.data.ConcatDataset</code> to form your new dataset. I believe your mistake is in separating the data from the labels, which should be in the same class. Here is a small example:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, data, target):
        self.data = data
        self.target = target

    def __getitem__(self, index):
        return self.data[index], self.target[index]

    def __len__(self):
        return len(self.data)

x = torch.rand(size=(666, 10), dtype=torch.float32)
x_labels = torch.zeros(size=(666,), dtype=torch.int32)
y = torch.rand(size=(80, 10), dtype=torch.float32)
y_labels = torch.ones(size=(80,), dtype=torch.int32)
z = torch.rand(size=(59, 10), dtype=torch.float32)
z_labels = torch.zeros(size=(59,), dtype=torch.int32)

x_dataset = CustomDataset(x, x_labels)
y_dataset = CustomDataset(y, y_labels)
z_dataset = CustomDataset(z, z_labels)

train_dataset = torch.utils.data.ConcatDataset([x_dataset, y_dataset, z_dataset])
print(train_dataset.cumulative_sizes)
</code></pre>
<p>Hope it helps!</p>

username: kwcckw
cooked: <p>Hi beaupreda,</p>
<p>Thanks and appreciate the information.</p>
<p>I didn’t aware of the inherit feature until you mention it, this sure looks useful.</p>
<p>Actually what i’m trying to do is perform segmentation classification with unet, so in each epoch, i would need to feed in a mask and an image. Do you aware of a better way to do this?</p>
<p>Anyway, i will proceed with your suggestion and see how it would work, thanks again</p>