Multiple Datasets

ahmed: <p>I have created a dataset class and dataloader for one of my datasets:</p>
<pre><code class="lang-auto">class Visual_DataSet(Dataset):
    def __init__(self, csv_path, root_dir):
        
        self.to_tensor = transforms.ToTensor()
        self.data_info = pd.read_csv(csv_path)
        self.root_dir = root_dir
        self.image_arr = np.asarray(self.data_info.iloc[:, 0])
        self.label_arr = np.asarray(self.data_info.iloc[:, 2])
        self.data_len = len(self.data_info.index)

    def __getitem__(self, index):
        single_image_name = self.image_arr[index]
        img_as_img = Image.open(self.root_dir+single_image_name)
        img_as_tensor = self.to_tensor(img_as_img)
        single_image_label = self.label_arr[index]
        return (img_as_tensor, single_image_label)

    def __len__(self):
        return self.data_len

if __name__ == "__main__":
    vs_trainset =  \
        DataSet(csv_path = 'images/visible/train_batch/trainset_batch.csv',
               root_dir = 'images/visible/train_batch/')
    vs_testset = \
        DataSet(csv_path = 'images/visible/test_batch/trainset_batch.csv',
               root_dir = 'images/visible/test_batch/')
</code></pre>
<pre><code>trainloader = torch.utils.data.DataLoader(dataset=fused_data,
                                                batch_size=4,
                                                shuffle=True)
testloader = torch.utils.data.DataLoader(dataset=fused_data,
                                                batch_size=4,
                                                shuffle=False)
</code></pre>
<p>I want to add another dataset (i am trying to fuse the datasets). So I have created another class for my other dataset in the same manner as the first previous one. Is there any way that I have put the two datasets together in a single dataloader? Thanks in advance.</p>

ptrblck: <p>You could provide all <code>Datasets</code> as a sequence to <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">ConcatDataset</a> to create a single one, which you can pass to the <code>DataLoader</code>.</p>

ahmed: <p>Thanks for the quick reply. I tried that previously:</p>
<pre><code class="lang-auto">class ConcatDataset(Dataset):
    
    def __init__(self, datasets):
        super(ConcatDataset, self).__init__()
        assert len(datasets) &gt; 0, 'datasets should not be an empty iterable'
        self.datasets = list(datasets)
        self.cumulative_sizes = self.cumsum(self.datasets)

    def __len__(self):
        return self.cumulative_sizes[-1]

    def __getitem__(self, idx):
        dataset_idx = bisect.bisect_right(self.cumulative_sizes, idx)
        if dataset_idx == 0:
            sample_idx = idx
        else:
            sample_idx = idx - self.cumulative_sizes[dataset_idx - 1]
        return self.datasets[dataset_idx][sample_idx]
</code></pre>
<pre><code class="lang-auto">fused_data = torch.utils.data.ConcatDataset(vs_trainset, th_trainset)
</code></pre>
<p>But I kept getting the error:</p>
<pre><code class="lang-auto">TypeError: __init__() takes 2 positional arguments but 3 were given
</code></pre>
<p>Can you see any obvious error that I may have overlooked?</p>

ptrblck: <p>Pass the <code>Datasets</code> as a list or tuple and it should work.</p>

ahmed: <p>It still gives me the same error:</p>
<pre><code class="lang-auto">fused_trainset = torch.utils.data.ConcatDataset([vs_trainset], [th_trainset])
</code></pre>
<pre><code class="lang-auto">TypeError: __init__() takes 2 positional arguments but 3 were given
</code></pre>

ptrblck: <p>Sorry for not being clear enough. You should pass one list containing all <code>Datasets</code>:</p>
<pre><code class="lang-python">fused_trainset = torch.utils.data.ConcatDataset([vs_trainset, th_trainset])
</code></pre>

ahmed: <p>Thank you so very much. It works now.</p>

Kylin9511: <p>Hi ptrblck~</p>
<p>I met a similar but more complicated scenario: I got two datasets A and B, and the batch_size_A was 30 while the batch_size_B was 60. ConcatDataset can only deal with the case where A and B are equivalent. So is there any official for imbalanced multi dataset sampling?</p>

ptrblck: <p>If you want to use different batch sizes for the datasets, you could either use different <code>DataLoader</code>s or probably create a custom sampler, which would use the predefined sample indices given your batch sizes.</p>

bigswede74: <p>I have built a custom Dataset class for loading COCO instance segmentation datasets. ConcatDataset is not available when inheriting from the abstract class Dataset but I have the need to merge multiple datasets. Any help would be appreciated.</p>
<p>My custom dataset is based off of this example.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://medium.com/fullstackai/how-to-train-an-object-detector-with-your-own-coco-dataset-in-pytorch-319e7090da5" target="_blank" rel="nofollow noopener" title="04:13PM - 09 April 2020">Medium – 9 Apr 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:129/86;"><img src="https://miro.medium.com/max/1200/0*VQgzrYnpo0jPcoH2" class="thumbnail" width="129" height="86"></div>

<h3><a href="https://medium.com/fullstackai/how-to-train-an-object-detector-with-your-own-coco-dataset-in-pytorch-319e7090da5" target="_blank" rel="nofollow noopener">How to train an Object Detector with your own COCO dataset in PyTorch</a></h3>

<p>Understand and apply PyTorch’s Dataset &amp; DataLoader to train an Object Detector with your own COCO formatted data</p>

  <p><span class="label1">Reading time: 4 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>