Image Folder with many non-class subfolders

mkutu: <p>Hi,<br>
I’m building a image classifier using trail camera data. I’m trying to create a custom dataset by loading images from a directory with many sub folders organized by capture location and date and not by classes.</p>
<p>for example,.</p>
<pre><code class="lang-auto">/data
    /COL
            /COLE
                        /CO_LE_05_190211
                                     /CLE-COLE05_00001_1-6-2019.jpg
                                     /CLE-COLE05_00001_1-25-2019.jpg
                                     .
                                     .

                        /CO_LE_05_190227
                                     /CLE-COLE05_00002_2-5-2019.jpg
                                     /CLE-COLE05_00002_2-6-2019.jpg
                                     .
                            
                        /CO_LE_05_190404
                                     /CLE-COLE05_00003_4-7-2019.jpg
                                     /CLE-COLE05_00003_4-13-2019.jpg
                                     .   
                                     .
</code></pre>
<p>I have a csv file with image paths and their respective label so I could take a subset of the dataset and structure sub-folders based on each class, however, doing so would limit accuracy, inferencing, etc.</p>
<p>I have over 3 million images and have found that re-organizing images by class would not be possible considering my memory limitations and time needed to do so.</p>
<p>I was wondering if anyone could suggest a custom dataset loader that would allow me to reference the csv with image path and label info, while retaining the current folder structure?</p>
<p>any help is greatly appreciated.</p>
<p>cheers,<br>
mkutu</p>

ptrblck: <p>I think the approach of using your already valid .csv file and keeping the folder structure sounds reasonable and I would avoid moving around the data.</p>
<p>To use the csv data, I would recommend to load it in your <code>Dataset.__init__</code> method with e.g. <code>pandas</code> via <code>pd.read_csv</code> and use the <code>index</code> in <code>Dataset.__getitem__(self, index)</code> to get the current file path and label.<br>
I’m not sure how your csv is structured, but you can index the rows and columns of the <code>pd.DataFrame</code> easily.<br>
Once you have the image path, you could load it with <code>PIL</code> and transform it to a tensor.<br>
The label will most likely be returned as a numpy array/scalar, so you could use <code>torch.from_numpy()</code> to create the target tensor.</p>

Omnia_Al-wazzan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I have a similar problem to <a class="mention" href="/u/mkutu">@mkutu</a></p>
<p>I have a breast cancer images dataset which is arranged as the following:</p>
<p>Data |---- ID_1 — 2016 —1-N  |1-1.dcm<br>
|---- ID_1 — 2016 —1-N |1-2.dcm<br>
|<br>
|---- ID_2 — 2016 —1-N  |1-1.dcm<br>
|---- ID_2 — 2016 —1-N  |1-2.dcm<br>
|<br>
|<br>
|---- ID_3 — 2016 —1-N  |1-1.dcm<br>
|---- ID_3 — 2016 —1-N  |1-2.dcm<br>
|---- ID_3 — 2016 —1-N  |1-3.dcm<br>
|---- ID_3 — 2016 —1-N  |1-4.dcm</p>
<p>And I have a CSV file that has the patient ID which is the same name as the first folder in the above format, for example:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e94cf51c0b69c58d05168e52cd48e6e0050f036.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e94cf51c0b69c58d05168e52cd48e6e0050f036" title="2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e94cf51c0b69c58d05168e52cd48e6e0050f036.png" alt="2" data-base62-sha1="4mx99RGd3DJKamsOPy25WPQ3V2e" width="690" height="80" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">750×88 3.15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would like to read the image along with its label from the csv, is that possible? I am confused about how to use the os.walk()  function inside my dataloader.</p>
<p>I would greatly appreciate it if you could point me in the right direction</p>
<p>Thanks</p>

ptrblck: <p>You wouldn’t implement any data loading loading into the <code>DataLoader</code> but inside your custom <code>Dataset</code>.<br>
The main idea would be the same as previously described:<br>
In the <code>Dataset.__init__</code> method you would store the paths to each sample by processing the CSV file.<br>
To do so you could read the CSV file via <code>pandas</code> and create the paths using the root folder etc.<br>
Once this is done you would then store all paths in e.g. a <code>list</code> and index it in the <code>__getitem__</code> to load each sample.</p>

Omnia_Al-wazzan: <p>As usual, your feedback is always constructive, thanks a lot.</p>
<p>Here I followed your suggestion:</p>
<pre><code class="lang-auto">
import numpy as np
import torch
from torchvision import transforms
import random
from pathlib import Path
#from typing import Tuple, Dict, List
import pydicom
import glob
import sys
import os
from torch.utils.data import Dataset,DataLoader
from pydicom.pixel_data_handlers.util import apply_voi_lut
import matplotlib.pyplot as plt
NUM_WORKERS=0
PIN_MEMORY=True    



transform_1 =  transforms.Compose([transforms.ToTensor()])

def read_xray(path, voi_lut = True, fix_monochrome = True):
    dicom = pydicom.read_file(path)
    
    # VOI LUT (if available by DICOM device) is used to transform raw DICOM data to "human-friendly" view
    if voi_lut:
        data = apply_voi_lut(dicom.pixel_array, dicom)
    else:
        data = dicom.pixel_array
               
    # depending on this value, MRI may look inverted - fix that:
    if fix_monochrome and dicom.PhotometricInterpretation == "MONOCHROME2":
        data = np.amax(data) - data
        
    data = data - np.min(data)
    data = data / np.max(data)
    data = (data * 255).astype(np.uint8)
        
    return data



class Dataset_(Dataset):
    def __init__(self, image_dir,transform=None):
        self.image_dir = image_dir
       # self.genes=path_genes_data
        self.images = list(image_dir.glob("*/*/*/*.dcm"))
        self.transform = transform

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        ## reading image ###
        img_path = os.path.join(self.image_dir, self.images[index])
        image = read_xray(img_path)
        #plt.figure(figsize = (12,12))
        #plt.imshow(image, 'gray')
        image_name=self.images[index]
        if self.transform is not None:
            a = self.transform(image=image)
            image = a['image']
            #image=np.transpose(image, (2, 0, 1))
            
        return image,self.images[index]



 
def Data_Loader( test_dir,batch_size,num_workers=NUM_WORKERS,pin_memory=PIN_MEMORY):
    
    load_data = Dataset_( image_dir=test_dir,transform=transform_1)

    data_loader = DataLoader(load_data,batch_size=batch_size,num_workers=num_workers,pin_memory=pin_memory,shuffle=True)
    
    return data_loader


data_path = Path(r"E:\IAAA_CMMD\manifest-1616439774456")
images_folder = data_path / "CMMD"
loader=Data_Loader(images_folder,4)
a=iter(loader)
a1=next(a)
</code></pre>
<p>Though, I can plot the images and see them, which might tell that the data is uploading,  however I get this error when trying to verify using a1 = next(a)</p>
<pre><code class="lang-auto">runfile('E:/IAAA_CMMD/untitled7.py', wdir='E:/IAAA_CMMD')
Traceback (most recent call last):

  File E:\IAAA_CMMD\untitled7.py:83 in &lt;module&gt;
    a1=next(a)

  File ~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py:530 in __next__
    data = self._next_data()

  File ~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py:570 in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File ~\anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py:49 in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File ~\anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py:49 in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File E:\IAAA_CMMD\untitled7.py:61 in __getitem__
    a = self.transform(image=image)

TypeError: __call__() got an unexpected keyword argument 'image'
</code></pre>
<p>Did I make something wrong in the transformation? I can’t tell where the error is <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><code>ToTensor</code> does not expect an <code>image</code> argument:</p>
<pre><code class="lang-python">transform = transforms.ToTensor()
img = transforms.ToPILImage()(torch.randn(3, 224, 224))

out = transform(img) # works
out = transform(image=img)
# TypeError: __call__() got an unexpected keyword argument 'image'
</code></pre>

Omnia_Al-wazzan: <p>Thanks a lot for your time and effort!</p>
<p>Great to know that.</p>
<p>It’s working now.</p>