Combine two dataloaders

username: sparshgarg23
cooked: <p>Given two datasets of length 8000 and 1480 and their corresponding train and validation loaders,I would like o create a new dataloader that allows me to  iterate through those loaders.</p>
<p>I tried using concatenate datasets as shown below</p>
<pre><code class="lang-auto">class custom_dataset(Dataset):
  def __init__(self,*data_sets):
    self.datasets=data_sets
  def __getitem__(self,i):
    return tuple(d[i] for d in self.datasets)
  def __len__(self):
    return min(len(d) for d in self.datasets)

</code></pre>
<pre><code class="lang-auto">new_dataset=custom_dataset(dataset,dataset1)
</code></pre>
<pre><code class="lang-auto">new_train_loader=DataLoader(new_dataset,batch_size=16,sampler=train_sampler,num_workers=2,drop_last=True)
new_val_loader=DataLoader(new_dataset,batch_size=16,sampler=val_sampler,num_workers=2,drop_last=True)
</code></pre>
<pre><code class="lang-auto">print("Training samples {},Val Samples {}".format(len(new_train_loader),len(new_val_loader)))
</code></pre>
<pre><code class="lang-auto">Training samples 74,Val Samples 18 this is the same number of samples for dataset 2
Training samples 400,Val Samples 100 for dataset 1
</code></pre>
<p>Since we are concatenating our dataset shouldn’t the final no of train samples be 474 and not 74?</p>
<p>But then when I print the lenght of the dataset it comes out to be 1480 and not 9400 why is that?<br>
If we concatenate two datasets having 8000 and 1480 samples shouldn’t the final dataset be of  length 9400 and not 1480.</p>
<p>Although I am able to visualize both samples from the two datasets,I somehow feel that a majority of the samples in the first dataset have not been included.<br>
Any suggestions on what is wrong with my following code</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sparshgarg23" data-post="1" data-topic="156164">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sparshgarg23/48/41060_2.png" class="avatar"> sparshgarg23:</div>
<blockquote>
<p>But then when I print the lenght of the dataset it comes out to be 1480 and not 9400 why is that?<br>
If we concatenate two datasets having 8000 and 1480 samples shouldn’t the final dataset be of length 9400 and not 1480.</p>
</blockquote>
</aside>
<p>You are not concatenating the datasets, but are returning tuples from both while using the <code>min(len())</code> of both datasets.<br>
If you want to concatenate both dataset (i.e. iterate through them in a sequential way) use <code>ConcatDataset</code>.</p>

username: sparshgarg23
cooked: <p>thanks for the suggestion,I will look into it<br>
I was able to make it work but it ends up returning only one image which could either belong to Dataset 1 or dataset 2.</p>
<p>Is there a way to combine my previous code with the  principles of concatDataset so that I can iterate through it in a sequential way and have it return tuples from both datasets.<br>
Meaning final output of concat dataset will be<br>
{image_1,label_1,image_2,label_2}</p>

username: ptrblck
cooked: <p>I don’t understand what the difference to your current implementation would be as you are currently already returning tuples and are thus not concatenating the datasets.<br>
Since both datasets might have a different length, you are using the <code>min</code> of them for the iteration, which sounds also right.<br>
E.g. assuming the first dataset has 2 samples, while the second one has 4, how should the tuple creation look in this case if you are not using the <code>min</code> length?</p>

username: sparshgarg23
cooked: <p>although the tuple creation is correct,when I initiate training only the first 74/400 samples are taken from the dataset 1.The rest of them are ignored.</p>
<p>This means that during training the train error decreases and the validation error increases.<br>
I would like to use all the training samples from both datasets and then pass them through the model.<br>
the reason why i need to iterate through both datasets is to compute feature similiarity between the two samples from dataset 1 and dataset 2.</p>
<p>I tried using iter as shown in the below implementation</p>
<pre><code class="lang-auto">len_source=len(train_loader_1)-1
len_target=len(val_loader)-1
iter_source=iter(train_loader_1)
iter_target=iter(val_loader)

for iter_num in range(epochs):
  optimizer.zero_grad()
  if iter_num%len_source==0:
    iter_source=iter(train_loader_1)
  if iter_num%len_target==0:
    iter_target=iter(val_loader)
  data_source=iter_source.next()
  data_target=iter_target.next()
  input_source,input_label=data_source
  input_tgt,label_tgt=data_target
  input_source=input_source.to(device)
  input_label=input_label.to(device)
  input_tgt=input_tgt.to(device)
  label_tgt=label_tgt.to(device)
  src_feature,src_output=model(input_source)
  tgt_feature,tgt_output=model(input_tgt)
</code></pre>
<p>This ends up working well,but I am not sure if all the batches are being taken into consideration or if it’s just one batch at each epoch.<br>
As per my understaning,if my epochs is set to 10 then I will end up calling iter.next will only consider 10 batches from each dataset</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sparshgarg23" data-post="5" data-topic="156164">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sparshgarg23/48/41060_2.png" class="avatar"> sparshgarg23:</div>
<blockquote>
<p>I would like to use all the training samples from both datasets and then pass them through the model.</p>
</blockquote>
</aside>
<p>How would you like to draw samples from both datasets if their lengths are unequal?<br>
Currently you are limiting the length to the <code>min</code> length of both datasets as already explained, which is one possibility. If you want to repeat the smaller dataset until all samples from the larger one are used, you could use a modulo operation in the indexing of the smaller dataset.</p>

username: sparshgarg23
cooked: <pre><code class="lang-auto">If you want to repeat the smaller dataset until all samples from the larger one are used, you could use a modulo operation in the indexing of the smaller dataset.
</code></pre>
<p>Could you let me know how to implement the modulo operation for indexing the smaller dataset.<br>
Should i use it in the training loop or in the custom dataset block?</p>

username: ptrblck
cooked: <p>You can use it in the <code>__getitem__</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.small_dataset = torch.arange(10).view(10, 1)
        self.large_dataset = torch.arange(20).view(20, 1)
        
    def __getitem__(self, index):
        a = self.small_dataset[index % len(self.small_dataset)]
        b = self.large_dataset[index]
        return a, b
    
    def __len__(self):
        return len(self.large_dataset)

dataset = MyDataset()
print(len(dataset))
# 20

loader = DataLoader(dataset, batch_size=2)

for idx, (a, b) in enumerate(loader):
    print("iter {}\na {}\nb {}".format(idx, a, b))
</code></pre>

username: jS5t3r
cooked: <p>can I somehow merge a and b in the class MyDataset?</p>
<p>a = (data_a, targets_a)<br>
b = (data_b, targets_b)</p>
<p>res = (data_a + data_b, targets_a + targets_b)</p>

username: ptrblck
cooked: <p>Assuming <code>data_x</code> and <code>targetx_x</code> are tensors you can concatenate them via <code>torch.cat</code> and pass the new tensor to the custom <code>Dataset</code>.</p>

username: jS5t3r
cooked: <p>I understand, I want to have exactly 50:50 from both dataset.</p>
<p>I did it in the for loop.</p>