Train simultaneously on two datasets

lcelona: <p>Hello,</p>
<p>I should train using samples from two different datasets, so I initialize two DataLoaders:</p>
<pre><code>train_loader_A = torch.utils.data.DataLoader(
             datasets.ImageFolder(traindir_A),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)

train_loader_B = torch.utils.data.DataLoader(
             datasets.ImageFolder(traindir_B),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)
</code></pre>
<p>What is the best way to extract samples from both iterators? In order to use something like this:</p>
<blockquote>
<p>for i, (input, target) in enumerate(train_loader):<br>
…</p>
</blockquote>
<p>Thanks.</p>

apaszke: <p>I’d recommend creating a new dataset and concatenating the images there, so the copy will be done inside the worker processes:</p>
<pre><code class="lang-python">class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)

train_loader = torch.utils.data.DataLoader(
             ConcatDataset(
                 datasets.ImageFolder(traindir_A),
                 datasets.ImageFolder(traindir_B)
             ),
             batch_size=args.batch_size, shuffle=True,
             num_workers=args.workers, pin_memory=True)

for i, (input, target) in enumerate(train_loader):
    ... 
</code></pre>

lcelona: <p>Perfect! Thank you for your help.</p>
<p>However, <code>__getindex__</code> should be <code>__getitem__</code>, correct?</p>

apaszke: <p>Yes, that’s a typo. Sorry. I’ve changed the code.</p>

Ahmed_Abbas: <p>Any idea on how to know, on the merged data whether the sample is coming from datasetA or datasetB in this case? I am trying to train a GAN-like network which needs to sample from two data sources and therefore needs the label in this case (A or B).</p>
<p>Sorry to wake-up an old thread</p>

platero: <p>I’ve read the code of <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataset.py#L42" rel="nofollow noopener"><code>ConcatDataset</code></a>. One question is that: how can we control the ratio of data coming from a specific dataset?</p>
<p>For example, dataset A contains 100 data, and dataset B contains 10000 data. How can we get 1:1 data from both A and B in one mini-batch? Does anybody have any idea? Thank you.</p>

cjb60: <p>To add to platero’s reply, suppose for example that datasetA contains 100 elements and datasetB contains 10000. My impression is that the data loader will (in one epoch) create shuffled indices 1…100 for datasetA and shuffled indices 1…100 for dataset B and create batches from each of those (since the <strong>len</strong> of ConcatDataset is the minimum of the lengths of both A and B). However, datasetB also has elements from 101…10000, so these will not be accessed. Am I correct in my intuition here? If so, this doesn’t seem like a reasonable solution if one dataset is way smaller than the other.</p>

mlopezantequera: <p>The current implementation does not discard data, it randomly samples from all the concatenated datasets. This means that each dataset will be sampled with probability <code>len(dataset)/len(all_datasets)</code></p>
<p>To change this behavior, I think one could use WeightedRandomSampler, setting the appropriate weights.</p>

mostafaaminnaji: <p>Hi Dear Adam<br>
I want to train my models simultaneously on two datasets, but I want to pick batches in the same order with shuffle=True. but targets1 and targets2 are not same. For example:</p>
<pre><code class="lang-auto">
train_dl1 = torch.utils.data.DataLoader(train_ds1, batch_size=16, 
                                       shuffle=True, num_workers=8)
train_dl2 = torch.utils.data.DataLoader(train_ds2, batch_size=16, 
                                       shuffle=True, num_workers=8)
inputs1, targets1 = next(iter(train_dl1))
inputs2, targets2 = next(iter(train_dl2))

targets1
tensor([ 1,  1,  0,  1,  0,  0,  1,  1])

targets2
tensor([ 0,  0,  0,  0,  0,  0,  0,  1])
</code></pre>
<p>I want to get <strong>targets1</strong> and <strong>targets2</strong> with the same order with the same shuffle. are you have any idea and can you help me?</p>

Arshiya_Aggarwal: <p>hey! this really helps but what if I have different samplers for both the data loaders? How do I train them simultaneously then?</p>

kli-nlpr: <p>The same with you, do you have any ideas? thank you</p>

PRAVEEN_KUMAR: <p>class ConcatDataset(Dataset):</p>
<pre><code>def __init__(self,*datasets):
    self.datasets = datasets
    self.data_files =os.listdir('../data/cat1/cat05')

def __getitem__(self, i):

    return tuple(d[i] for d in self.datasets)

def __len__(self):
    return  min(len(d) for d in self.datasets)
</code></pre>
<p><span class="hashtag-raw">#def</span> loadImgs(des_dir = “…/data/”,img_size=128,batchSize = 4):</p>
<p>dataset1 = dset.ImageFolder(root=“…/data/cat1/”,<br>
transform=transforms.Compose([<br>
transforms.Resize(128),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),<br>
]))<br>
dataset2 = dset.ImageFolder(root=“…/data/cat2/”,<br>
transform=transforms.Compose([<br>
transforms.Resize(128),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),<br>
]))<br>
#<br>
<span class="hashtag-raw">#print</span>(dataset)<br>
dataloader = torch.utils.data.DataLoader(<br>
ConcatDataset(dataset1,dataset2),<br>
batch_size= 4, # how many samples per batch to load<br>
shuffle=True)</p>
<p>dataloadercat1 = torch.utils.data.DataLoader(<br>
dataset1,<br>
batch_size= 4, # how many samples per batch to load<br>
shuffle=True)</p>
<p><span class="hashtag-raw">#inputs1</span>,targets1 = next(iter(dataset1))<br>
<span class="hashtag-raw">#inputs2</span>,targets2 = next(iter(dataset2))</p>
<p>for epoch in range(1):<br>
for i,(inputs,lables) in enumerate(dataloader):</p>
<pre><code>    print("Batch all ",i,lables)
for i, (inputs, lables) in enumerate(dataloadercat1):
    print("Batch cat ", i)
print(ConcatDataset(dataset1,dataset2).datasets)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/ba6c3b0ab7574a8aef512e82bffad7402210b4ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba6c3b0ab7574a8aef512e82bffad7402210b4ff" title="folderstructure"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/ba6c3b0ab7574a8aef512e82bffad7402210b4ff.png" alt="folderstructure" data-base62-sha1="qBaGWdken9meckMnLuNV5m8SPdZ" width="395" height="500" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">folderstructure</span><span class="informations">468×592 3.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

PRAVEEN_KUMAR: <p>After implementation ConcatDataset with over data set than only smaller size subfolder is consider in batch formation. this reason some image of other category is missed out.   you have any idea how to solved out this problem.?</p>

PRAVEEN_KUMAR: <p>when i have developed this concept in my dataset than at the time of batch formation only smaller size data is consider in batch formation.<br>
e.g  data folder i have two subfolders cat1 and cat2. in cat1  70 images and cat2 30 images . Dataloader function make only 30 image considered in batch formation. 40 images of cat2 is not considered in batch formation.? can you have any solution which consider remaning image also</p>

Haris_Cheong: <p>Hi there, I have managed to use two datasets by creating a custom dataset that takes in two root directories:</p>
<pre><code class="lang-auto">class dataset_maker(Dataset):
    def __init__(self, root_dir1, root_dir2, transform= None):
        self.root_dir1=root_dir1
        self.root_dir2=root_dir2
        self.filelist1 = glob.glob(root_dir1+'*.png')
        self.filelist2 = glob.glob(root_dir2+'*.png')
        self.transform=transform
        
    def __len__(self):
        return min(len(self.filelist1),len(self.filelist2))
    
    def __getitem__(self, idx):
        sample1 = io.imread(self.filelist1[idx])/65535*255
        sample2 = io.imread(self.filelist2[idx])/65535*255
        sample1=np.uint8(sample1)
        sample2=np.uint8(sample2)
        sample1=PIL.Image.fromarray(sample1)
        sample2=PIL.Image.fromarray(sample2)
        if self.transform:
            sample1 = self.transform(sample1)
            sample2 = self.transform(sample2)
        return sample1,sample2
</code></pre>
<p>then, make a dataloader using the two datasets:</p>
<pre><code class="lang-auto">dataloader = DataLoader(combined_dataset, batch_size=3, shuffle=True, num_workers=4)
</code></pre>
<p>Finally, I get the data in the training loops by doing this call in the for loop:</p>
<pre><code class="lang-auto">for epoch in range(10):
    running_loss=0.0
    
    #get the data
    for batch_num, (hq_batch,Lq_batch) in enumerate(dataloader):
        print(batch_num, hq_batch.shape, Lq_batch.shape)
</code></pre>
<p>The output is stated below:</p>
<pre><code class="lang-auto">0 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
1 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
2 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
3 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
4 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
5 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
6 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
7 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
8 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
9 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
10 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
11 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
12 torch.Size([3, 3, 256, 256]) torch.Size([3, 3, 256, 256])
</code></pre>
<p>Hope this solves the problem!</p>

Sumesh_Uploader: <p>how this combined_dataset is to be designed? Is that to be instance of Concat. I got this error while tried to form the combined dataset using dset.ImageFolder, :“TypeError: expected str, bytes or os.PathLike object, not ConcatDataset”</p>

miladiouss: <p>I’m not sure if the for loop is setup correctly since <code>__getitem__</code> is returning <code>( data points for A, labels for A) , (data points for B, labels for B) )</code>.</p>

Vij: <p>How do you train your model ?<br>
You had passed batch_size =3 in the dataloader, but you got two batches of 3 sample. How do you use it to train model?</p>

Vij: <p>Did you find the solution for this? I.e when samplers are not of same size?<br>
Another question,. Can we use concat dataset for more than 2 samplers?</p>

vishalthengane: <p>Hi There,</p>
<p>Suppose I have two training dataset with different size and I am trying to train it on a network simultaneously, So I can do it? also, I need to keep a track of from which dataset image is coming to find out the loss after each iteration by the equation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4ddf90b8012a122933379c0553b2cfccb759a4de.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ddf90b8012a122933379c0553b2cfccb759a4de" title="Screenshot%20from%202019-03-19%2010-50-30"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4ddf90b8012a122933379c0553b2cfccb759a4de.png" alt="Screenshot%20from%202019-03-19%2010-50-30" data-base62-sha1="b6TKBu78d5z3Td7AQmiaF0MjSdU" width="690" height="100" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-03-19%2010-50-30</span><span class="informations">1020×148 8.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where,<br>
L0 &amp; L1 are the lengths of the dataset and Lambda is a balancing constant.</p>
<p>Thank you.</p>