How ConcatDataset work?

username: Nikolaos_Sintoris
cooked: <p>Hello everyone.</p>
<p>I have 3 folders (for example A, B, C) and then every folder has 3 subfolders with the same name (for example sub1, sub2, sub3). Every subfolder contains images and basically every subfolder represents a different class.</p>
<p>I want to create a training data set with these 3 folders and I found ConcatDataset that might work.</p>
<p>So first of all I created 3 different train_datasets using ImageFolder and then I created a list with these 3 train_datasets (lets say that this list has the name all_datasets).<br>
Then in order to create my final training dataset, i used concatdataset</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">A_dataset = torchvision.datasets.ImageFolder(root = A_directory , transform = transform)
B_dataset = torchvision.datasets.ImageFolder(root = B_directory , transform = transform)
C_dataset = torchvision.datasets.ImageFolder(root = C_directory , transform = transform)

all_datasets = []
all_datasets.append(A_dataset)
all_datasets.append(B_dataset)
all_datasets.append(C_dataset)

final_training_dataset = torch.utils.data.ConcatDataset(all_datasets)
</code></pre>
<p>Can anyone explain to me how the format of final_training_dataset is?<br>
I am afraid that it will confuse my classes and thus, I will not have the right labels.<br>
Is there a problem or is everything fine?</p>
<p>Thank you everyone.</p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/nikolaos_sintoris">@Nikolaos_Sintoris</a><br>
I tried with a simple implementation of the code you have provided.</p>
<p>The <code>ConcatDataset</code> has a <code>.datasets</code> attribute that lists all the datasets that are concatenated. This helps in the demarcation of all the labels. You would need to iterate on all the datasets in the <code>ConcatDataset</code> each having its own set of labels.</p>
<p>Hope this helps you.</p>
<h2>
<a name="code-to-reproduce-1" class="anchor" href="#code-to-reproduce-1"></a>Code to reproduce</h2>
<pre><code class="lang-bash">$ mkdir A B C
$ mkdir A/sub_1 A/sub_2 A/sub_3
$ mkdir B/sub_1 B/sub_2 B/sub_3
$ mkdir C/sub_1 C/sub_2 C/sub_3
</code></pre>
<p>Creating random images in each sub folder</p>
<pre><code class="lang-python">for folder in ["A", "B", "C"]:
    for sub_folder in os.listdir(folder):
        for i in range(10):
            img = np.random.random((20,20))
            plt.imsave(arr=img, fname=f"{folder}/{sub_folder}/img_{i}.png")
</code></pre>
<p>Using the code snippet provided by you</p>
<pre><code class="lang-python">A_dataset = torchvision.datasets.ImageFolder(root = "A" , transform = torchvision.transforms.ToTensor())
B_dataset = torchvision.datasets.ImageFolder(root = "B" , transform = torchvision.transforms.ToTensor())
C_dataset = torchvision.datasets.ImageFolder(root = "C" , transform = torchvision.transforms.ToTensor())

all_datasets = []
all_datasets.append(A_dataset)
all_datasets.append(B_dataset)
all_datasets.append(C_dataset)

final_training_dataset = torch.utils.data.ConcatDataset(all_datasets)
</code></pre>
<p>The sanity check</p>
<pre><code class="lang-python">for dataset in final_training_dataset.datasets:
    print(dataset.root)
    print(dataset.classes)
</code></pre>
<p>Result:</p>
<pre><code class="lang-nohighlight">A
['sub_1', 'sub_2', 'sub_3']
B
['sub_1', 'sub_2', 'sub_3']
C
['sub_1', 'sub_2', 'sub_3']
</code></pre>

username: Nikolaos_Sintoris
cooked: <p>Ok, so if I create a dataloader from the final_training_dataset I am ok.</p>
<pre><code class="lang-auto">trainloader = torch.utils.data.DataLoader(final_training_dataset, batch_size = 1, shuffle = True)  

for image, label in trainloader:
      #process my images
</code></pre>
<p>Am I right?</p>

username: ariG23498
cooked: <p>With the <code>trainloader</code>, the labels would only correspond to <code>0</code>, <code>1</code> and <code>2</code>. This will create a problem, you would not be able to distinguish from which subfolder the image comes from.</p>
<p>To iterate upon the problem, you have three folders and three sub-folders in each of them. This means that you have in total <code>3*3=9</code> individual classes.</p>
<p>I would suggest you to do the following:</p>
<ul>
<li>Make sure the classes of every <code>&lt;&gt;_dataset</code> is distinct.<pre data-code-wrap="python"><code class="lang-python">for ind, c in enumerate(A_dataset.classes):
    A_dataset.classes[ind] = f"A_{c}"

for ind, c in enumerate(B_dataset.classes):
    B_dataset.classes[ind] = f"B_{c}"

for ind, c in enumerate(C_dataset.classes):
    C_dataset.classes[ind] = f"C_{c}"
</code></pre>
</li>
<li>ConcatDataset holds the order in which the datasets are provided unless the <code>shuffle=True</code> parameter is present. If you keep a tab of the counts of data you could see from which dataset the data comes from.</li>
</ul>
<p>A minimal implementation can be seen here in this <a href="https://gist.github.com/ariG23498/af76b2b0b2c59cb6eb9aaf90fa75793d" rel="noopener nofollow ugc">GitHub gist</a>.</p>
<blockquote>
<p>If the gist does not load, please use <a href="https://nbviewer.jupyter.org/gist/ariG23498/af76b2b0b2c59cb6eb9aaf90fa75793d" rel="noopener nofollow ugc">nbviewer</a></p>
</blockquote>

username: Nikolaos_Sintoris
cooked: <p>Yes but fortunately i do have only 3 classes. I mean that my subfolders represent the same 3 classes in every folder. So it is not a problem that i will only have 0, 1, 2 labels. I just wanted to make sure that in folder A, subfolder1 has label 0, subfolder2 has label 1 and subfolder3 has label 2. The same thing happens for the other folders (for example in folder B, subfolder1 has label 0, subfolder2 has label 1 etc).<br>
So eventually i do not have problem with trainloader right? It will not confuse the labels. In every folder, always subfolder1 will have label 0, right?</p>

username: ariG23498
cooked: <p>Yes. Then it should not be any problem. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: Nikolaos_Sintoris
cooked: <p>Thanks for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>