Load Data and Train simultaneously on two datasets

mostafaaminnaji: <p>Hello Friends<br>
I want to train my models simultaneously on two datasets, but I want to pick batches in the same order with shuffle=True. but targets1 and targets2 are not same. For example:</p>
<pre><code class="lang-auto">
train_dl1 = torch.utils.data.DataLoader(train_ds1, batch_size=8, 
                                       shuffle=True, num_workers=8)
train_dl2 = torch.utils.data.DataLoader(train_ds2, batch_size=8, 
                                       shuffle=True, num_workers=8)
inputs1, targets1 = next(iter(train_dl1))
inputs2, targets2 = next(iter(train_dl2))
</code></pre>
<p>but</p>
<pre><code class="lang-auto">targets1
tensor([ 1,  1,  0,  1,  0,  0,  1,  1])

targets2
tensor([ 0,  0,  0,  0,  0,  0,  0,  1])
</code></pre>
<p>I want to get targets1 and targets2 with the same order with the same shuffle. Are you have any idea and can you help me?</p>

justusschock: <p>I guess you could use <a href="https://pytorch.org/docs/master/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">ConcatDataset</a> for this.</p>
<p>Sorry I’m currently typing from my mobile phone and thus I am unable to provide some sample code</p>

mostafaaminnaji: <p><a class="mention" href="/u/justusschock">@justusschock</a> Dear Justus, thanks for your response.<br>
Yes, I know what is  “ContactDataset”. But it mixes all Dataset Randomly.<br>
Actually, I want to train two network which each network should train on the special dataset and finally I want to use their result to another network (For ensemble learning) while I should keep the shuffle.<br>
In another word, I want to “targets1” and “targets2” be same.<br>
Thanks again for your answer, my friend.</p>
<pre><code class="lang-auto">train_dl1 = torch.utils.data.DataLoader(train_ds1, batch_size=8,  shuffle=True, num_workers=8)                                 
train_dl2 = torch.utils.data.DataLoader(train_ds2, batch_size=8,                                      shuffle=True, num_workers=8)
inputs1, targets1 = next(iter(train_dl1))
inputs2, targets2 = next(iter(train_dl2))
targets1
tensor([ 1,  1,  0,  1,  0,  0,  1,  1])
targets2
tensor([ 0,  0,  0,  0,  0,  0,  0,  1])
</code></pre>

ptrblck: <p>Are both datasets identical or different?<br>
If they are identical and you would like to sample in a random but defined order, you could use a <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py#L57">SubsetRandomSampler</a>, where you can define your own indices, for both <code>DataLoader</code>s.</p>

justusschock: <p>I’m sorry that I didn’t understand your requirements clearly.</p>
<p>Another approach could be to define your own Dataset which returns two tuples instead of one (one per dataset each with data and label) in the <code>__getitem__</code> method. This way you could define the way, the data processing and indexing is done. I will try to provide some sample code with my mobile phone. I cannot guarantee it to work, but it should give you an idea how to do it.</p>
<p>EDIT: Dataset example code:</p>
<pre><code class="lang-auto">class DoubleDataset(Dataset):
    
    def __init__(self, path_1, path_2, transforms=None):
        self.data_1 = [os.path.join(path_1, x) for x in os.listdir(path_1) if x.endswith(".png")]

        self.data_2 = [os.path.join(path_2, x) for x in os.listdir(path_2) if x.endswith(".png")]

        self.transform = transforms

    def __getitem__(self, index):
        _data1 = custom_load_fn(self.data_1[index])
        _data1 = custom_load_fn(self.data_2[index])

        if self.transforms is not. None:
            _data1 = self.transforms(_data1)
            _data2 = self.transforms (_data2) 

        return _data1, _data2

    def __len__(self):
        return min(len(self.data_1), len(self.data_2))
</code></pre>
<p>This way you could train your networks like:</p>
<pre><code class="lang-auto">for data1, data2 in dataloader:
    pred1 = model1(data1[0])
    pred2 = model2(data2[0])

    optim1.zero_grad()
    loss1 = loss_fn(pred1, data1[1])
    loss1.backward()
    optim1.step()

    optim2.zero_grad()
    loss2 = loss_fn(pred2, data2[1])
    loss2.backward()
    optim2.step()
</code></pre>
<p>If setting <code>shuffle=True</code> the data loader gives a random index to the dataset and the dataset then returns items of both datasets with the same index</p>

qiminchen: <p>Hi,</p>
<p>did you already solve this problem? How? thanks</p>

saba: <p>Hi Ptrblck,</p>
<p>I want to use 2 different dataset but simultaneously. I used</p>
<pre><code class="lang-auto"> for ii , data in enumerate (trainloader)
~~~
to get my data but now I want to get another data from another dataloadr in same time same batch size.
is it correct to use:

~~~ 
   images1, targets1 = next(iter(trainloader))      
   images2, targets2 = next(iter(trainloaderNeg))   

   ~~~~</code></pre>

ptrblck: <p>The last two lines of code would recreate the <code>iterator</code>, so you should split the calls to:</p>
<pre><code class="lang-python">data_iter1 = iter(trainloader)
data_iter2 = iter(trainloaderNeg)

images1, targets1 = next(data_iter1)
images2, targets2 = next(data_iter2)
...
</code></pre>
<p>Note that you would have to catch the <code>StopIteration</code> once the <code>iterators</code> are empty.</p>

saba: <p>many thanks for your help.<br>
Sorry if I want to multiply to dataset (a and b)  with the size of  (64,1,21,21), the torch.mul(a,b) does work properly?</p>

ptrblck: <p>Which error are you getting and what are the shapes of <code>a</code> and <code>b</code>?</p>

saba: <p>Did not give me any error, just I want to know the way of multiplication is safe/<br>
a is the batch of positive patches with size of 21x 21 and b is the batch of negative patch with the same size</p>

ptrblck: <p>I’m not familiar with your use case, but</p>
<pre><code class="lang-python">a = torch.randn(6, 1, 21, 21)
b = torch.randn(6, 1, 21, 21) 
torch.mul(a, b)
</code></pre>
<p>would perform an element-wise multiplication.</p>

saba: <p>Hi Ptrblck,</p>
<p>I hope you are well. Sorry for GAN , I nned to compute the SWD distance I found this link (“<a href="https://github.com/koshian2/swd-pytorch/blob/master/README.md" rel="nofollow noopener">https://github.com/koshian2/swd-pytorch/blob/master/README.md</a>”)<br>
Do you think the code is reliable to use?</p>
<p>I appreciate you if suggest me a link for FID score too. I found another link but I am not sure if they are reliable to use or not?</p>
<p>Cheers<br>
Saba</p>

ptrblck: <p>Unfortunately, I haven’t used this repository, but would recommend to just try it out, run some quick tests, and make sure your dummy examples return the expected outputs.</p>

saba: <p>Sorry, for using the FID score in GAN. Do you think is it good enough to implement the metric with Numpy and compute the FID score?</p>
<p>Or the idea is worthy when we use the pre-trained inception model?</p>

Ikram_Hattab: <p>Hi Guys,</p>
<p>so, I want to train my model on two datasets (RGB and thermal images) <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7254f43c36f63bb2deff9799f15d900f6e842ecb.png" alt="Capture" data-base62-sha1="gjqrzVAtIEZmGRCmwwfUqW8CUGL" width="192" height="259">  , and I want to pick batches in the same order with shuffle=True.</p>
<p>i already have a function create_dataloader :</p>
<p>def create_dataloader(path, imgsz, batch_size, stride, opt, hyp=None, augment=False, cache=False, pad=0.0, rect=False,</p>
<pre><code>                  rank=-1, world_size=1, workers=8, image_weights=False, quad=False, prefix='', shuffle=True):

# Make sure only the first process in DDP process the dataset first, and the following others can use the cache

with torch_distributed_zero_first(rank):

    dataset = LoadImagesAndLabels(path, imgsz, batch_size,

                                  augment=augment,  # augment images

                                  hyp=hyp,  # augmentation hyperparameters

                                  rect=rect,  # rectangular training

                                  cache_images=cache,

                                  single_cls=opt.single_cls,

                                  stride=int(stride),

                                  pad=pad,

                                  image_weights=image_weights,

                                  prefix=prefix,

                                  shuffle=shuffle)

batch_size = min(batch_size, len(dataset))

nw = min([os.cpu_count() // world_size, batch_size if batch_size &gt; 1 else 0, workers])  # number of workers

sampler = torch.utils.data.distributed.DistributedSampler(dataset) if rank != -1 else None

loader = torch.utils.data.DataLoader if image_weights else InfiniteDataLoader

# Use torch.utils.data.DataLoader() if dataset.properties will update during training else InfiniteDataLoader()



dataloader = loader(dataset,

                    batch_size=batch_size,

                    num_workers=nw,

                    sampler=sampler,

                    pin_memory=True,

                    collate_fn=LoadImagesAndLabels.collate_fn4 if quad else LoadImagesAndLabels.collate_fn)

return dataloader, dataset
</code></pre>
<p>i’m trying to create 2 data loader for each datasets like that :</p>
<h1><a name="trainloader-1" class="anchor" href="#trainloader-1"></a>Trainloader</h1>
<pre><code>dataloader1, dataset1 = create_dataloader("../Flir", imgsz, batch_size, gs, opt,

                                        hyp=hyp, augment=True, cache=opt.cache_images, rect=opt.rect, rank=rank,

                                        world_size=opt.world_size, workers=opt.workers, 

                                        image_weights=opt.image_weights, quad=opt.quad, prefix=colorstr('train: '), shuffle=True)

dataloader2, dataset2 = create_dataloader("../FlirRGB", imgsz, batch_size, gs, opt,

                                        hyp=hyp, augment=True, cache=opt.cache_images, rect=opt.rect, rank=rank,

                                        world_size=opt.world_size, workers=opt.workers,

                                        image_weights=opt.image_weights, quad=opt.quad, prefix=colorstr('train: '), shuffle=True)
</code></pre>
<p>But i didn’t have a right solution.<br>
can you help me please.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d9e8aae76d5dd5038432a0b3fc02102e22a64be.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d9e8aae76d5dd5038432a0b3fc02102e22a64be" title="Capture (1)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d9e8aae76d5dd5038432a0b3fc02102e22a64be.png" alt="Capture (1)" data-base62-sha1="kcOZQ38YIP15YVwOBHwAakZa250" width="690" height="301" data-dominant-color="474744"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture (1)</span><span class="informations">894×391 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I think the proper approach would be to write a custom <code>Dataset</code> (e.g. as given in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>) and load the images from both folders simultaneously. This would make sure to get the desired image pairs in the same order without trying to use the seeds etc.<br>
Let me know, if this approach would work.</p>

Ikram_Hattab: <p>Not yet.<br>
I have problem in datasets.</p>
<p>I split my dataset to train ,val and test and I already have 3716 images for train.<br>
So when i train my dataset “FlirRGB” (!python train.py --img 640 --batch 10 --epochs 12 --data FlirRGB.yaml --weights yolov3.pt) the system found only 2112 images (and 0 missing).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43b912cc817ac6c59fed7d0718c959318ce91ac6.png" alt="Capture" data-base62-sha1="9F6vwKVSC6dlub81MlMShtR6c7A" width="627" height="351"><br>
I have already jpg format.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1baa9e63e416a1bccbfb5f5f084eeae771288fe2.png" data-download-href="https://discuss.pytorch.org/uploads/default/1baa9e63e416a1bccbfb5f5f084eeae771288fe2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1baa9e63e416a1bccbfb5f5f084eeae771288fe2_2_690x387.png" alt="image" data-base62-sha1="3WKrjGLuGcMyky2YauXwGIozvB8" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1baa9e63e416a1bccbfb5f5f084eeae771288fe2_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1baa9e63e416a1bccbfb5f5f084eeae771288fe2_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1baa9e63e416a1bccbfb5f5f084eeae771288fe2.png 2x" data-dominant-color="383333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 218 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So please how i can solve my problem.</p>

ptrblck: <p>I’m not sure how you’ve created the <code>Dataset</code>, but I assume you are using a custom implementation.<br>
If that’s the case, note that the <code>__len__</code> method of the <code>Dataset</code> returns the number of samples, which can be drawn from this dataset.<br>
If you expect the <code>train_dataset</code> to have a length of <code>3716</code>, while only <code>2112</code> samples are returned, could you check the return value of <code>print(len(train_dataset))</code> and see, how what the <code>__len__</code> method is returning (i.e. how the length is calculated)?</p>

Ikram_Hattab: <p>I have sorted the contents of trainRGB.txt and It worked.</p>