Questions about Dataloader and Dataset

chenyuntc: I run my dataloader like this: <SCODE>dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))
 
dataloader=t.utils.data.DataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)
<ECODE> <SCODE>        try:
            img = self.loader(os.path.join(self.root, path))
        except Exception as e:
            index = index - 1 if index > 0 else index + 1 
            return self.__getitem__(index)
<ECODE>

smth: 

apaszke: 

chenyuntc: At first I try: <SCODE>def my_loader(path):
    try: 
        return Image.open(path).convert('RGB')
    except Exception as e:
        print e
def my_collate(batch):
    "Puts each data field into a tensor with outer dimension batch size"
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)

dataset= ImageFolder('/home/x/train/',
            transform=transforms.Compose([transforms.ToTensor()]),
            loader = my_loader)
dataloader=t.utils.data.DataLoader(dataset,4,True,collate_fn=my_collate)
<ECODE> so then I try this： <SCODE>def my_collate(batch):
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)
class MyImageFolder(ImageFolder):
    __init__ = ImageFolder.__init__
    def __getitem__(self, index):
        try: 
            return super(MyImageFolder, self).__getitem__(index)
        except Exception as e:
            print e

dataset= MyImageFolder('/home/x/train/', transform = transforms.Compose([transforms.ToTensor()..]) )
dataloader=t.utils.data.DataLoader(dataset, 4, True, collate_fn=my_collate)
<ECODE>

PermanentPon: To solve this particular problem with corrupted images you can just add two lines before your code: <SCODE>from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
<ECODE>

Sun_ShiJie: Hi, Chen. Does the batch size decrease when use this way to filter invalid data.

msamogh: Using nonechucks, your code would look something like this: <SCODE>dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))

import nonechucks as nc
dataset = nc.SafeDataset(dataset)
dataloader = nc.SafeDataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)

# You can now use `dataloader` as though it was a regular DataLoader without
# having to worry about the bad samples!
<ECODE>

nimning: How you get the internal function ‘default_collate’? It is within the ‘dataloder.py’.

tom: <SCODE>import torch
import torch.utils.data
print(torch.utils.data.dataloader.default_collate)
<ECODE> Best regards Thomas

zanaa: Note: for python 3, replace batch = filter(lambda x : x is not None, batch) with batch = list(filter(lambda x : x is not None, batch))

Shivam_Chandhok: I am getting an error: What am I doing wrong.

weiwei: 

Martin36: Is there a way to get this to work when the batch size is only 1?

Keyv_Krmn: 

xu555: from torch.utils.data.dataloader import default_collate

Syzygianinfern0: Did you find any fix?