Import Dataset from Computer

username: PyTorch2603
cooked: <p>Hi all.<br>
I was trying to load a dataset composed of a large number of images which are located on my computer (This PC/Desktop/mydata), where mydata is a folder containing the images. I also need to split the images into 80% training and 20% testing, while splitting training into 80% training and 20% validation.<br>
What would be the code needed for me to do this?<br>
Thanks for the help!</p>

username: ptrblck
cooked: <p>I don’t know how the images are stored and how you would like to use them, but if you are working on a multi-class classification and the images are stored in “class folders” (i.e. each folder contains images for a specific class), you could use the <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html#torchvision.datasets.ImageFolder"><code>ImageFolder</code></a> class.<br>
However, if this does not fit your use case, you might want to implement a custom <code>Dataset</code> as described e.g. <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.<br>
Once you’ve written this dataset, you could then use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split"><code>torch.utils.data.random_split</code></a> to split this dataset into a training and validation set.</p>

username: PyTorch2603
cooked: <p>Thank you for the reply, I will try to do this. I think I will use ImageFolder because there are certain categories, and in each category there are a set of images which belong to it.</p>

username: ptrblck
cooked: <p>That sounds indeed like a good fit for <code>ImageFolder</code> then.</p>

username: PyTorch2603
cooked: <p>Would I be able to use ImageFolder in the following fashion?<br>
import torchvision.datasets<br>
mydataset=torchvision.datasets.ImageFolder(root=“This PC/Desktop/mydata”)<br>
If so I am still unsure how I would use this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7bc0fcdf66d767315c01f226d1188c39d62f230.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7bc0fcdf66d767315c01f226d1188c39d62f230" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7bc0fcdf66d767315c01f226d1188c39d62f230.png" alt="image" data-base62-sha1="zlyR2YyBw4Qw2tpqRbrfaGaLBfi" width="690" height="49" data-dominant-color="E7E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">842×61 4.97 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My question being how would I be able to specify that this splits into training and testing, and then use random_split once more to split training into training and validation. I do not see any way that I am specifying that one section is training and the other is testing (or validation).<br>
Could you provide some code and/or an explanation as to how I would do this, as well as any changes I have to make to my usage of ImageFolder? Thanks!</p>

username: ptrblck
cooked: <p>The <code>root</code> path looks wrong and you should specify it as a path that Python can find (if this path is working on a Windows system, ignore my advice as I’m not really familiar with Windows setups).</p>
<p>To split the data, you could use:</p>
<pre><code class="lang-python">nb_samples = len(dataset)
print(nb_samples)
# 100

train_split = int(nb_samples*0.8)

train_dataset, val_test_dataset = torch.utils.data.random_split(dataset, [train_split, nb_samples-train_split])
print(len(train_dataset), len(val_test_dataset))
# 80 20

val_split = int(len(val_test_dataset)*0.5)
val_dataset, test_dataset = torch.utils.data.random_split(val_test_dataset, [val_split, len(val_test_dataset)-val_split])
print(len(val_dataset), len(test_dataset))
# 10 10
</code></pre>
<p>If you need a more advanced method to create the splits, you could also check e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.test_train_split</code></a>, create the indices for each split, and use <code>torch.utils.data.Subset</code> to create the training, validation, and test datasets.</p>

username: PyTorch2603
cooked: <p>I understand the splitting portion now, thank you!<br>
It is a Windows OS but the root path was incorrect, like you said. It is actually Desktop/mydata.<br>
Also, to get my desired distribution, it would be 0.64 training, 0.16 validation, and 0.2 testing (since I am splitting it 80/20 training/test, but then within training splitting 80/20 training/validation), so I could just change the 0.8 to 0.64, and the 0.5 to 4/9 and it would work, correct?<br>
Thanks for the help, I really appreciate it.</p>

username: ptrblck
cooked: <p>Yes, my example just used an 80-10-10 split, but you can of course change these values for your use case.</p>