Questions about ImageFolder

yichuan9527: Which data format of the picture that ImageFolder can read?  (pickle, JPG or PNG)

ncullen93: <SCODE>IMG_EXTENSIONS = [
    '.jpg', '.JPG', '.jpeg', '.JPEG',
    '.png', '.PNG', '.ppm', '.PPM', '.bmp', '.BMP',
]
<ECODE>

yichuan9527: 

yichuan9527: 

xiahouzuoxin: <SCODE>def find_classes(dir):
classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
classes.sort()
class_to_idx = {classes[i]: i for i in range(len(classes))}
return classes, class_to_idx<ECODE>

ncullen93: The labels are the sub-folders from the main directory. Say you have mnist images separated by digit like this: <SCODE>main_dir/
     0/
         img1_digit0.jpg
         img2_digit0.jpg
     1/
         img3_digit1.jpg
         ....
     ....
     9/
        ...
<ECODE> <SCODE>from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
data = ImageFolder(root='main_dir', transform=ToTensor())
<ECODE> Now the unique set of class labels is found easily, but this isn’t the class label for each individual image. <SCODE>print(data.classes) # ['0','1',..'9']
<ECODE> And the images can be accessed  by an integer index: <SCODE>x,y = data[0] # x is the first image as PIL, y is that images class label
<ECODE> You can easily iterate through these images: <SCODE>for i in range(len(data)):
    x,y = data[i]
<ECODE> <SCODE>from torch.utils.data import DataLoader
loader = DataLoader(data)
for x, y in loader:
    print(x) # image
    print(y) # image label
<ECODE> <SCODE>loader = DataLoader(data, shuffle=True)
<ECODE> <SCODE>from torch.utils.data import RandomSampler
sampler = RandomSampler(data)
loader = DataLoader(data, sampler=sampler)
<ECODE> <SCODE>import numpy as np
def my_numpy_loader(filename):
    return np.load(filename)

data = ImageFolder(root='path/to/numpy/maindir/', loader=my_numpy_loader, transform=ToTensor())
<ECODE> Example: <SCODE>import numpy as np
def my_numpy_loader(filename):
    return np.load(filename)
from torchvision.transforms import Compose, ToPILImage, ToTensor, ...
my_transform = Compose([ToPILImage(), ... typical transforms .., ToTensor()])
data = ImageFolder(root='path/to/numpy/maindir/', 
                loader=my_numpy_loader, transform=my_transform)
<ECODE> Here, you convert numpy -> pil image -> do some transforms -> convert back to torch tensor…

Veril: 

ncullen93: 

yichuan9527: Dose pytorch have any functions to read the images from pickle file?

apaszke: 

FuriouslyCurious: I noticed that ImageFolder can process following structure: <SCODE>root/dog/xxx.png
root/dog/xxy.png

root/cat/123.png
root/cat/asd932_.png
<ECODE> My question is, can it process a deeper /nested folder structure? e.g.: <SCODE>root/mammal/dog/xxx.png
root/mammal/cat/xxy.png

root/avian/pigeon/123.png
root/avian/crow/nsdf3.png
<ECODE> If yes, would the classes be “mammal, avian”, or would they be “dog, cat, pigeon, crow”? Thanks!

chenyuntc: <SCODE>class TrainImageFolder(ImageFolder):

    def __getitem__(self, index):
            filename = self.imgs[index]#  'avian/crow/nsdf3.png'
            if filename.split('/')[0] =='avian': label = 0
            ### or 
            # if filename.split('/')[1] =='dog': label = 0
            return super(TrainImageFolder, self).__getitem__(index)[0],  label
<ECODE>

Stefan_Radonjic: 

ptrblck: 

Stefan_Radonjic: I have a small dataset of only 1200 images. My idea was to to load dataset, perform lets say Random Crop on it, then load dataset again but this time perform ColorJitter and then merge those two datasets into one which will result in dataset of 2400 pictures. I wanted to create a dataset of atleast 7200 pictures ( which is 6 transformations each time i load dataset) . Do you by any chance have a better idea?

ptrblck: <SCODE>class MyData(Dataset):
    def __init__(self):
        self.data = torch.randn(100)
        
    def __getitem__(self, index):
        return self.data[index]
    
    def __len__(self):
        return len(self.data)
    
data1 = MyData()
data_loader1 = DataLoader(data1, batch_size=1)

for batch_idx, data in enumerate(data_loader1):
    print(batch_idx) # prints to idx 99
    
data2 = MyData()
concat_data = ConcatDataset((data1, data2))
concat_loader = DataLoader(concat_data, batch_size=1)

for batch_idx, data in enumerate(concat_loader):
    print(batch_idx) # prints to idx 199
    
print(len(data_loader1)) # 100
print(len(concat_loader)) # 200
<ECODE> I cannot see line 89. Would you please post the code snippet?

Sumesh_Uploader: 

ptrblck: <SCODE>class MyDataset(Dataset):
    def __init__(self, dataset1, dataset2):
        self.dataset1 = dataset1
        self.dataset2 = dataset2
        
    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]
        
        return x1, x2
    
    def __len__(self):
        return len(self.dataset1)


dataset1 = TensorDataset(torch.randn(100, 1))
dataset2 = TensorDataset(torch.randn(100, 1))
dataset = MyDataset(dataset1, dataset2)
<ECODE>

nimning: Hi, Nick Is there way I can edit the IMG_EXTENSIONS so that the ImageFolder would be able to read more image with .gif and .oct-stream extension? Best

Alan_Wu: Then I suppose it’s that the class dog is encoded into [1, 0] as for the first category, And the cat goes to [0, 1], Am I right?