Combine / concat dataset instances

bodokaiser: <p>What is the recommended approach to combine two instances from <code>torch.utils.data.Dataset</code>?</p>
<p>I came up with two ideas:</p>
<ol>
<li>Wrapper-Dataset:</li>
</ol>
<pre><code class="lang-python">class Concat(Dataset):

    def __init__(self, datasets):
        self.datasets = datasets
        self.lengths = [len(d) for d in datasets]
        self.offsets = np.cumsum(self.lengths)
        self.length = np.sum(self.lengths)

    def __getitem__(self, index):
        for i, offset in enumerate(self.offsets):
            if index &lt; offset:
                if i &gt; 0:
                    index -= self.offsets[i-1]
                return self.datasets[i][index]
        raise IndexError(f'{index} exceeds {self.length}')

    def __len__(self):
        return self.length
</code></pre>
<ol start="2">
<li>Using <code>itertools.chain</code>
</li>
</ol>
<pre><code class="lang-python">loader = itertools.chain(*[MyDataset(f'file{i}') for i in range(1, 4)])
</code></pre>

apaszke: <p>Doesn’t <code>itertools.chain</code> return an iterator that will go over them only once? I think there’s no recommended way, it’s all down to a personal taste <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

bodokaiser: <p>Ah yes <code>itertools.chain</code> would only do one epoch so we would be better of with something like:</p>
<pre><code class="lang-python">x = itertools.repeat(itertools.chain.from_iterable([dataset1, dataset2]), times=epochs)

next(next(iter(x))

# or 

for epoch in x:
    for (inputs, targets) in epoch:
       print(inputs)
</code></pre>

apaszke: <p>Not sure if that’s going to work. It can break if <code>itertools.chain</code> iterator is not immutable (and it’s probably not). It would be simpler to do this (or use the wrapper dataset):</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    dset = itertools.chain(...)
    dloader = # create DataLoader
    for ... in dloader:
        ...
</code></pre>

acgtyrant: <p>I can not pass the <code>itertools.chain</code> instance to <code>torch.utils.data.DataLoader</code> class while the former does not support <code>len()</code>.</p>

jnhwkim: <p>See <code>ConcatDataset</code> updated in v0.3: <a href="http://pytorch.org/docs/master/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">http://pytorch.org/docs/master/data.html#torch.utils.data.ConcatDataset</a></p>

kavyajeetbora: <pre><code class="lang-auto">def return_all_items(dataset):
  all_items = []
  for i in range(len(dataset)):
    all_items.append(dataset[i])
  return all_items
</code></pre>
<pre><code class="lang-auto">list1 = return_all_items(original_data)
list2 = return_all_items(transformed_data)
list1.extend(list2)
</code></pre>
<p>then converting the list to dataset object</p>
<pre><code class="lang-auto">class AugmentedDataset(Dataset):
  
  def __init__(self, combined_list, transform=None):
    self.combined_list = combined_list
    
  def __len__(self):
    return len(self.combined_list)
  
  def __getitem__(self, idx):
    sample = self.combined_list[idx]      
    return sample
  
augmented_dataset = AugmentedDataset(list1)
</code></pre>

jasha: <p>See also <code>ChainDataset</code>: <a href="https://pytorch.org/docs/master/data.html#torch.utils.data.ChainDataset" rel="nofollow noopener">https://pytorch.org/docs/master/data.html#torch.utils.data.ChainDataset</a></p>