How does ConcatDataset work?

username: zimmer550
cooked: <p>Hello. This is my CustomDataSetClass:</p>
<pre><code class="lang-auto">class CustomDataSet(Dataset):
    def __init__(self, main_dir, transform):
        self.main_dir = main_dir
        self.transform = transform
        all_imgs = os.listdir(main_dir)
        self.total_imgs = natsort.natsorted(all_imgs)
        for file_name in self.total_imgs:
            if '.txt' in file_name: self.total_imgs.remove(file_name)
            if file_name == 'semantic': self.total_imgs.remove('semantic')

    def __len__(self):
        return len(self.total_imgs)

    def __getitem__(self, idx):
        img_loc = os.path.join(self.main_dir, self.total_imgs[idx])
        image = Image.open(img_loc).convert("RGB")
        tensor_image = self.transform(image)
        return tensor_image
</code></pre>
<p>Here is how I create a list of datasets:</p>
<pre><code class="lang-auto">    all_datasets = []
    while folder_counter &lt; num_train_folders:
        #some code to get path_to_imgs which is the location of the image folder
        train_dataset = CustomDataSet(path_to_imgs, transform)
        all_datasets.append(train_dataset)
        folder_counter += 1
</code></pre>
<p>Then I concat my datasets and create the dataloader and do the training:</p>
<pre><code class="lang-auto">final_dataset = torch.utils.data.ConcatDataset(all_datasets)
train_loader = data.DataLoader(final_dataset,
                                   batch_size=batch_size,
                                   shuffle=False,
                                   num_workers=0,
                                   pin_memory=True,
                                   drop_last=True)
</code></pre>
<p>So, is the order of my data preserved? During training, will I go to each folder in theexact order that the concatenation was done and then grab all the images sequentially? For example:</p>
<p>I grab 150 images from folder 1, 100 images from folder 2 and 70 images from folder 3. I concatenate my the three datasets. During training I do:</p>
<pre><code class="lang-auto">for idx, input_seq in enumerate(data_loader):
#code to train
</code></pre>
<p>So, will the dataloader go through folder 1 and grab all the images inside there sequentially and then go to folder 2 and do the same and finally go to folder 3 and do the same as well? I tried reading the code for ConcatDataset but I can’t understand whether the order of my data willl be preserved or not.</p>

username: ptrblck
cooked: <p>Yes, the order should be preserved as shown in this simple example using <code>TensorDatasets</code>:</p>
<pre><code class="lang-python">datasets = []
for i in range(3):
    datasets.append(TensorDataset(torch.arange(i*10, (i+1)*10)))

dataset = ConcatDataset(datasets)
loader = DataLoader(
    dataset,
    shuffle=False,
    num_workers=0,
    batch_size=2
)

for data in loader:
    print(data)
</code></pre>

username: mohammed_guermal
cooked: <p>what if I want to grab data from different file but after i want it to be concatinated and shuffled not preserving the same order<br>
because I have data in different folders so i grabe each of them but afterwards i want it to be all shuffled.</p>

username: ptrblck
cooked: <p>You could use <code>shuffle=True</code> when creating the <code>DataLoader</code>, which will shuffle the passed <code>ConcatDataset</code>.</p>

username: nisharaichur
cooked: <pre><code class="lang-auto">list_1 = [1,2,3,4,5]
list_2 = [6,7,8,9,10]
list_3 = [22,23,24,25,26,27]
dataset_list = [list_1, list_2, list_3]

dataset_loader = DataLoader(dataset_list, shuffle=True, batch_size=3)
for i in range(30):
    for x in dataset_loader:
        print(x)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92cb4ba858ed2ba1d53cf616b09e415c228b8a05.png" data-download-href="https://discuss.pytorch.org/uploads/default/92cb4ba858ed2ba1d53cf616b09e415c228b8a05" title="output"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92cb4ba858ed2ba1d53cf616b09e415c228b8a05.png" alt="output" data-base62-sha1="kWBgHrrBPh2BfCGia9qejgBFD2B" width="690" height="453" data-dominant-color="343534"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">output</span><span class="informations">789×518 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My question is why does each batch(size 3) has same data but shuffled within themselves?<br>
My usecase is, I need to shuffle the entire dataset after concatenation, such that over each epoch I have different batch of dataset shuffled over all the datasets[list_1, list_2, list_3]</p>

username: ptrblck
cooked: <p>Passing nested lists to the <code>DataLoader</code> might have these kind of side effects and thus I would recommend to create tensors, pass them to a <code>TensorDataset</code>, and this dataset to the <code>DataLoader</code>, which should then properly index and shuffle the data.</p>

username: nisharaichur
cooked: <p>Sure Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: adwaykanhere
cooked: <p>Hi,<br>
I’m trying to use ConcatDataset to concat the training and testing sets of the CIFAR10 dataset.<br>
For my application, I also need to combine the dataset.target values for each set. But from my understanding, I cannot derive combined targets using ConcatDataset.<br>
Please suggest how to go about concatenating the datasets such that I can get targets as well.</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/5">here</a>.</p>

username: Brando_Miranda
cooked: <p>hmmm but for real data sets each of them will have their own data transforms e.g. their own mean, std to normalize the images. How do you deal with that? I’d assume this is dealt at the each data set level by giving them the right transforms when instantiating the data set pytorch object.</p>
<hr>
<p>I think this: <a href="https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362" class="inline-onebox">Concat image datasets with different size and number of channels</a> answers my question. Which my suggestion was right.</p>

username: Brando_Miranda
cooked: <p>how did you solve your target labels issue? <a class="mention" href="/u/adwaykanhere">@adwaykanhere</a></p>
<hr>
<p>this is key: <a href="https://discuss.pytorch.org/t/does-concatenate-datasets-preserve-class-labels-and-indices/62611/3" class="inline-onebox">Does Concatenate Datasets preserve class labels and indices - #3 by ptrblck</a></p>
<hr>
<p>easiest solution to what I want is to do use this: <a href="https://discuss.pytorch.org/t/does-concatenate-datasets-preserve-class-labels-and-indices/62611/12" class="inline-onebox">Does Concatenate Datasets preserve class labels and indices - #12 by Brando_Miranda</a> by using learn2learn’s union of data sets.</p>