ChainDataset + random_split oddity

aarrvv: <p>My training data is split into multiple files so I have to use ConcatDataset, then I split them into train/test/valid sets using random_split.</p>
<p>I don’t think this is a very uncommon situation but I cannot get this to work since one wants an iteratable dataset and the other wants an indexable dataset, so I end up with something the works in neither situation. Both</p>
<p><code>next(iter(train_ds))</code></p>
<p>and</p>
<p><code>train_ds[0]</code></p>
<p>give me this error:</p>
<pre><code class="lang-auto">NotImplementedError                       Traceback (most recent call last)
&lt;ipython-input-22-de92b112278f&gt; in &lt;module&gt;
----&gt; 1 next(iter(train_ds))

~/.local/lib/python3.6/site-packages/torch/utils/data/dataset.py in __getitem__(self, idx)
    255 
    256     def __getitem__(self, idx):
--&gt; 257         return self.dataset[self.indices[idx]]
    258 
    259     def __len__(self):

~/.local/lib/python3.6/site-packages/torch/utils/data/dataset.py in __getitem__(self, index)
     23 
     24     def __getitem__(self, index):
---&gt; 25         raise NotImplementedError
     26 
     27     def __add__(self, other):

NotImplementedError: 
</code></pre>
<p>Switching to ConcatDataset gives me this:</p>
<p><code>AssertionError: ConcatDataset does not support IterableDataset</code></p>
<p>What Am I doing wrong?</p>

aarrvv: <p>Should add that I am using a custom IteratableDataset that also implements indexing.</p>
<p>Unfortunately I cannot fix this by joining the raw files and loading them as one since the data is interpreted as sequences and joining them would create invalid sequences right where two files meet.</p>

ptrblck: <p>It’s only 9 months later <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"><br>
But in any case: could you describe how <code>IterableDataset</code>s should be concatenated? They don’t have a length, since they are used for e.g streaming data, so I would assume that you could just switch the stream once it’s exhausted?</p>

jsetty: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I want to use 2 <code>IterableDatasets</code> for each epoch. As you said, once the iterator reaches the end of the first <code>IterableDataset</code>, I want to use the second ‘IterableDataset’. How do I achieve this?</p>

ptrblck: <p>You can just continue with the second <code>IterableDataset</code> as seen here:</p>
<pre><code class="lang-python">class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, start, end):
        super(MyIterableDataset).__init__()
        assert end &gt; start, "this example code only works with end &gt;= start"
        self.start = start
        self.end = end
        
    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:  # single-process data loading, return the full iterator
            iter_start = self.start
            iter_end = self.end
        else:  # in a worker process
            # split workload
            per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
            worker_id = worker_info.id
            iter_start = self.start + worker_id * per_worker
            iter_end = min(iter_start + per_worker, self.end)
        return iter(range(iter_start, iter_end))


ds1 = MyIterableDataset(start=3, end=7)
loader1 = DataLoader(ds1, num_workers=1)
ds2 = MyIterableDataset(start=7, end=10)
loader2 = DataLoader(ds2, num_workers=1)

for epoch in range(2):
    for data in loader1:
        print(data)
        # train with first loader
        
    for data in loader2:
        print(data)
        # train with second loader
</code></pre>

jsetty: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> How is it different compared to <code>ChainDataset</code>? If they are the same, why was <code>ChainDataset</code> not your first recommendation?</p>

ptrblck: <p><code>ChainDataset</code> would also work and you can pick any approach that fits your needs.<br>
I don’t know how the initial request would work with <code>IterableDataset</code>s as they are apparently streaming on one hand but also support indexing. The latter would be needed for the splits I would assume, but also the author didn’t follow up so I don’t know how it was solved.</p>

jsetty: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your quick answer!</p>