Using only some of Subfolders in dataset

username: Blueberry
cooked: <p>I have a dataset folder which consists of four sub-folders [say - f1,f2,f3,f4] each with images from different class. How can only read three of specified folders say  [f1,f2,f4] for image classification problem? Currently I am reading all the folders by default using</p>
<pre><code class="lang-auto">data = datasets.ImageFolder(train_dir,transform=transform)
train_loader = torch.utils.data.DataLoader(data,batch_size=batch_size,sampler=train_sampler)
</code></pre>

username: ptrblck
cooked: <p>You could create a new folder with symbolic links to the desired 3 folders or alternatively move the unwanted folder to another place.</p>

username: Blueberry
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I was working on a huge kaggle dataset so I wanted to know if there was a way I can only read three out of four subfolders for classes in Pytorch.</p>

username: ptrblck
cooked: <p>Symbolic links won’t move any data. Would that work for you?</p>

username: Blueberry
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please elaborate on what do you mean by that and how can that be done.</p>

username: ptrblck
cooked: <p><code>ln -s source_dir destination_dir</code> will create a symbolic link of the source directory (or file) to the specified destination without moving any files.<br>
If you would like to use only a few subfolders, you could create a new root folder and then create symbolic links to the desired image folders inside the new root folder.<br>
This will make sure that <code>ImageFolder</code> only sees the desired class folders instead of all.</p>
<p>PS: Assuming you use a Linux OS.</p>

username: saba
cooked: <p>Hi Ptrblck,</p>
<p>I am facing the same problem “/home/work/” includes many folders and  in "dataroot = ‘/home/work/radius_32_32/’ I have 4 more folders that include my  data to use. Where I should add the link line in the code to use only radius_32_32 and subfolders in it?</p>
<p>" ln -s /home/work/  /home/work/radius_32_32"</p>
<pre><code class="lang-auto">dataroot = '/home/work/radius_32_32/'

dataset = dset.ImageFolder(root=dataroot,
                            transform=transforms.Compose([
                                transforms.Resize(image_size),
                                transforms.CenterCrop(image_size),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                            ]))

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="saba" data-post="7" data-topic="72323">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saba/48/19439_2.png" class="avatar"> saba:</div>
<blockquote>
<p>to use only radius_32_32 and subfolders in it</p>
</blockquote>
</aside>
<p>You don’t need to create symbolic links and can just pass <code>/home/work/radius_32_32</code> to your <code>ImageFolder</code>, if you want to use this folder as the root and the subfolders as the class folders.</p>

username: Shyam_Gupta196
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
can you suggest something for windows,<br>
I was using cats vs dogs dataset and it turns out i only want to use dogs subfolder .</p>
<p>What should i do so that imagefolder only sees dog folder ,</p>
<p>I have data in read only mode and cannot be changed ,<br>
hence i cannot move it to other folder ,</p>
<p>pls help me out</p>

username: ptrblck
cooked: <p>I believe Windows can also create symbolic links (they might have a different name?) so you should be able to use the same workflow in creating a new folder with the sym links to the desired data folder(s).</p>

username: Kaykay
cooked: <p>You can make your <code>batch_size=1</code> and skip everything other than specific classes during the training loop. Maybe it’s not the most efficient but it’s very easy:</p>
<pre><code class="lang-auto">classes: list[str] = sorted(os.listdir(datadir))
targets: list[int] = [classes.index(label) for label in classes if label in ('Dog','Cat')]

for epoch in tqdm(range(epochs)):
	for images, labels in tqdm(trainloader):
		if not any(x in labels for x in targets):
			continue
</code></pre>

username: danK
cooked: <p>I had a very similar issue to the original post where I wanted to ignore certain classes within my root directory. After some tinkering I came up with the following solution that works well for me. The solution is based on subclassing the <code>DatasetFolder</code> class and overriding the <code>find_classes</code> method with a slight modification that ignores all folders specified in <code>dropped_classes</code>. The class can be interchangeably used with the standard <code>ImageFolder</code> class:</p>
<pre><code class="lang-auto">from torchvision import datasets
IMG_EXTENSIONS = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp')

class ClassSpecificImageFolder(datasets.DatasetFolder):
    def __init__(
            self,
            root,
            dropped_classes=[],
            transform = None,
            target_transform = None,
            loader = datasets.folder.default_loader,
            is_valid_file = None,
    ):
        self.dropped_classes = dropped_classes
        super(ClassSpecificImageFolder, self).__init__(root, loader, IMG_EXTENSIONS if is_valid_file is None else None,
                                                       transform=transform,
                                                       target_transform=target_transform,
                                                       is_valid_file=is_valid_file)
        self.imgs = self.samples

    def find_classes(self, directory):
        classes = sorted(entry.name for entry in os.scandir(directory) if entry.is_dir())
        classes = [c for c in classes if c not in self.dropped_classes]
        if not classes:
            raise FileNotFoundError(f"Couldn't find any class folder in {directory}.")

        class_to_idx = {cls_name: i for i, cls_name in enumerate(classes)}
        return classes, class_to_idx
</code></pre>