TarinLoader can be used for two datsets Pytorch?

username: Daniel_Joseph
cooked: <p>How to use the Dataloader to load two different datasets like :</p>
<pre><code class="lang-auto">
train_set = DataCustom(path=path, train=True)
train_loader = torch.utils.data.DataLoader(dataset=(train_set1, train_set2),
                                           batch_size=args.batch_size,
                                           pin_memory=True,
                                           shuffle=True,
                                           )

test_set = DataCustom(path=path, train=False)
test_loader = torch.utils.data.DataLoader(dataset=(test_set1,test_set2)
                                          batch_size=args.batch_size,
                                          pin_memory=True,
                                          shuffle=False,
                                          )
</code></pre>
<p>instead of writing two different dataloader like :</p>
<pre><code class="lang-auto"># load the First datasets 
train_set = DataCustom(path=path, train=True)
train_loader = torch.utils.data.DataLoader(dataset=train_set,
                                           batch_size=args.batch_size,
                                           pin_memory=True,
                                           shuffle=True,
                                           )
test_set = DataCustom(path=path, train=False)
test_loader = torch.utils.data.DataLoader(dataset=test_set,
                                          batch_size=args.batch_size,
                                          pin_memory=True,
                                          shuffle=False,
                                          )

# Load the second datasets 
train_set_2 = DataCustom(path=path_2, train=True)
train_loader_2 = torch.utils.data.DataLoader(dataset=train_set,
                                           batch_size=args.batch_size,
                                           pin_memory=True,
                                           shuffle=True,
                                           )

test_set_2 = DataCustom(path=path_2, train=False)
test_loader_2 = torch.utils.data.DataLoader(dataset=test_set,
                                          batch_size=args.batch_size,
                                          pin_memory=True,
                                          shuffle=False,
                                          )

</code></pre>
<p>Thanks in Advance (<a class="mention" href="/u/ptrblck">@ptrblck</a> for you special thanks dude <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> )</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">ConcatDataset</a> is probably what you want.</p>
<p>PS: I’m not a fan of tagging, as this might discourage others to post an answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Daniel_Joseph
cooked: <p>Yeah you are right , just because you are the one who answered all my questions so far , i though its gonna the same , that’s all.</p>

username: Daniel_Joseph
cooked: <p>I tried to do somehow like this :</p>
<pre><code class="lang-auto">
def train(epoch):
    for batch_idx, (data, target), (data_2, target_2) in enumerate(train_loader, train_loader_2):
        if use_cuda:
            data, target = data.cuda(), target.cuda()
            data_2,  target_2= data_2.cuda(), target_2.cuda()

        data, target = Variable(data), Variable(target)
        data_2,  target_2 = Variable(data_2), Variable(target_2)

        optimizer.zero_grad()

        data = data.float()
        data_2 = data_2.float()
        output = model(data, data_2 )

        prec1, = accuracy(output.data, target.data)
        loss = criterion(output, torch.max(target, 1)[0])
        loss.backward()
        optimizer.step()
</code></pre>
<p>Since both data have same target, the accyracy will remain the same, i kept using two DataLoader for both datasets,  i m wondering if there is anyway to do so ?</p>

username: ptrblck
cooked: <p>If you are using <code>shuffle=True</code> as it seems to be the case, the <code>target</code> tensors will most likely be different.</p>
<p>I misunderstood your use case, as I thought you would like to extend the dataset and call the additional samples in a sequential way.<br>
Could you explain the correspondence between both datasets, i.e. is <code>train_set[0]</code> corresponding to <code>train_set_2[0]</code>?<br>
If so, I would suggest to implement a custom <code>Dataset</code> and return the pair for a single index.<br>
Here is a small example:</p>
<pre><code class="lang-python">

class MyDataset(Dataset):
    def __init__(self, dataset1, dataset2):
        self.dataset1 = dataset1
        self.dataset2 = dataset2
        
    def __getitem__(self, index):
        x1, y1 = self.dataset1[index]
        x2, y2 = self.dataset2[index]
        
        if (y1 != y2).any():
            raise RuntimeError('ERROR! Target mismatch')
            
        return x1, x2, y1
        
    def __len__(self):
        return len(self.dataset1)  # Assuming both have same length
</code></pre>

username: Daniel_Joseph
cooked: <p>How can i use the enumirate for the two dastasets ?</p>

username: ptrblck
cooked: <p>You would wrap it in additional parentheses:</p>
<pre><code class="lang-python">for idx, ((data1, target1), (data2, target2)) in enumerate(zip(loader1, loader2)):
    print(data1.shape)
    print(data2.shape)
</code></pre>

username: Daniel_Joseph
cooked: <p>Thanks a lot, that’s what I was looking for,  thousands thumbs <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>