Train simultaneously on two datasets

lcelona: Hello, I should train using samples from two different datasets, so I initialize two DataLoaders: <SCODE>train_loader_A = torch.utils.data.DataLoader(
             datasets.ImageFolder(traindir_A),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)

train_loader_B = torch.utils.data.DataLoader(
             datasets.ImageFolder(traindir_B),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)
<ECODE> What is the best way to extract samples from both iterators? In order to use something like this: Thanks.

apaszke: I’d recommend creating a new dataset and concatenating the images there, so the copy will be done inside the worker processes: <SCODE>class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)

train_loader = torch.utils.data.DataLoader(
             ConcatDataset(
                 datasets.ImageFolder(traindir_A),
                 datasets.ImageFolder(traindir_B)
             ),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)

for i, (input, target) in enumerate(train_loader):
    ... 
<ECODE>

lcelona: Perfect! Thank you for your help.

apaszke: Yes, that’s a typo. Sorry. I’ve changed the code.

Ahmed_Abbas: Any idea on how to know, on the merged data whether the sample is coming from datasetA or datasetB in this case? I am trying to train a GAN-like network which needs to sample from two data sources and therefore needs the label in this case (A or B). Sorry to wake-up an old thread

platero: For example, dataset A contains 100 data, and dataset B contains 10000 data. How can we get 1:1 data from both A and B in one mini-batch? Does anybody have any idea? Thank you.

cjb60: 

mlopezantequera: To change this behavior, I think one could use WeightedRandomSampler, setting the appropriate weights.

mostafaaminnaji: <SCODE>
train_dl1 = torch.utils.data.DataLoader(train_ds1, batch_size=16, 
                                       shuffle=True, num_workers=8)
train_dl2 = torch.utils.data.DataLoader(train_ds2, batch_size=16, 
                                       shuffle=True, num_workers=8)
inputs1, targets1 = next(iter(train_dl1))
inputs2, targets2 = next(iter(train_dl2))

targets1
tensor([ 1,  1,  0,  1,  0,  0,  1,  1])

targets2
tensor([ 0,  0,  0,  0,  0,  0,  0,  1])
<ECODE>

Arshiya_Aggarwal: hey! this really helps but what if I have different samplers for both the data loaders? How do I train them simultaneously then?

kli-nlpr: The same with you, do you have any ideas? thank you

PRAVEEN_KUMAR: class ConcatDataset(Dataset): <SCODE>def __init__(self,*datasets):
    self.datasets = datasets
    self.data_files =os.listdir('../data/cat1/cat05')

def __getitem__(self, i):

    return tuple(d[i] for d in self.datasets)

def __len__(self):
    return  min(len(d) for d in self.datasets)
<ECODE> <SCODE>    print("Batch all ",i,lables)
for i, (inputs, lables) in enumerate(dataloadercat1):
    print("Batch cat ", i)
print(ConcatDataset(dataset1,dataset2).datasets)
<ECODE>

PRAVEEN_KUMAR: After implementation ConcatDataset with over data set than only smaller size subfolder is consider in batch formation. this reason some image of other category is missed out.   you have any idea how to solved out this problem.?

PRAVEEN_KUMAR: 

Haris_Cheong: Hi there, I have managed to use two datasets by creating a custom dataset that takes in two root directories: <SCODE>class dataset_maker(Dataset):
    def __init__(self, root_dir1, root_dir2, transform= None):
        self.root_dir1=root_dir1
        self.root_dir2=root_dir2
        self.filelist1 = glob.glob(root_dir1+'*.png')
        self.filelist2 = glob.glob(root_dir2+'*.png')
        self.transform=transform
        
    def __len__(self):
        return min(len(self.filelist1),len(self.filelist2))
    
    def __getitem__(self, idx):
        sample1 = io.imread(self.filelist1[idx])/65535*255
        sample2 = io.imread(self.filelist2[idx])/65535*255
        sample1=np.uint8(sample1)
        sample2=np.uint8(sample2)
        sample1=PIL.Image.fromarray(sample1)
        sample2=PIL.Image.fromarray(sample2)
        if self.transform:
            sample1 = self.transform(sample1)
            sample2 = self.transform(sample2)
        return sample1,sample2
<ECODE> then, make a dataloader using the two datasets: <SCODE>dataloader = DataLoader(combined_dataset, batch_size=3, shuffle=True, num_workers=4)
<ECODE> Finally, I get the data in the training loops by doing this call in the for loop: <SCODE>for epoch in range(10):
    running_loss=0.0
    
    #get the data
    for batch_num, (hq_batch,Lq_batch) in enumerate(dataloader):
        print(batch_num, hq_batch.shape, Lq_batch.shape)
<ECODE> The output is stated below: <SCODE>0 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
1 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
2 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
3 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
4 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
5 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
6 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
7 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
8 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
9 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
10 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
11 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
12 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
<ECODE> Hope this solves the problem!

Sumesh_Uploader: how this combined_dataset is to be designed? Is that to be instance of Concat. I got this error while tried to form the combined dataset using dset.ImageFolder, :“TypeError: expected str, bytes or os.PathLike object, not ConcatDataset”

miladiouss: 

Vij: 

Vij: 

vishalthengane: Hi There, Thank you.