Composing `Dataset`-s

username: franchesoni
cooked: <p>Abstract question:</p>
<p><strong>Can I compose custom Datasets and expect everything to work fine with no surprises?</strong><br>
Something like</p>
<pre><code class="lang-python">dat1 = Dat1()
dat2 = Dat2(dat1)
dat3 = Dat3(dat2)
dataloader = data.DataLoader(dat3)
</code></pre>
<p>where datasets are defined as</p>
<pre><code class="lang-python">class Dat3(data.Dataset):
  def __init__(self, dat2, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.dat2 = dat2
  
  def __get_item__(self, index):
    var2 = dat2.__getitem__(index)
    out = operation(var2)
    return out

</code></pre>
<hr>
<p>Particular question:<br>
I am implementing an interactive image segmentation application and I need to load the data in roughly three steps:</p>
<ol>
<li>Load one image and its masks of some segmentation dataset and return them</li>
<li>Select one target region from between the masks and return it</li>
<li>Sample points from inside a target region and return them<br>
finally:</li>
<li>Create a dataloader that loads image, target region and points</li>
</ol>
<p>My idea was to apply the “composition over inheritance” principle, which means creating 1. <code>SegDataset(data.Dataset)</code>, 2. <code>RegionDataset(data.Dataset)</code> which I init as <code>RegionDataset(seg_dataset, *args, **kwargs)</code> and 3. <code>IISDataset(data.Dataset)</code> which can be instantiated as <code>IISDataset(region_dataset, *args, **kwargs)</code>.</p>
<p>Will this work well if I use the last one to init a <code>DataLoader</code>?<br>
The difference between doing this and making custom datasets (not inheriting from <code>data.Dataset</code>) is only the inheritance, which may be good (I can dataload any intermediate dataset) but also bad (maybe I should be careful when doing distributed training). Any opinion?</p>

username: franchesoni
cooked: <p>I found that if I don’t inherit from <code>data.Dataset</code> the <code>DataLoader</code> still works. What is the advantage of subclassing <code>data.Dataset</code>?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="franchesoni" data-post="2" data-topic="145118">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/franchesoni/48/32282_2.png" class="avatar"> franchesoni:</div>
<blockquote>
<p>What is the advantage of subclassing <code>data.Dataset</code>?</p>
</blockquote>
</aside>
<p>Deriving from <code>Dataset</code> would raise errors e.g. if mandatory methods are missing (in your example you are using <code>__geti_item__</code> instead of <code>__getitem__</code> which should fail) and might thus be a cleaner approach.</p>
<p>Your “composition” approach should also work, but I would probably just call into the nested datasets directly instead of calling the <code>__geittem__</code> (<code>self.dat2[index]</code> instead of <code>self.dat2.__getitem__(index)</code>).</p>

username: nivek
cooked: <p>You can also try using <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc"><code>torchdata</code></a>, but it is still in prototype stage (soon to be in Beta).</p>
<p>The tutorial provides <a href="https://pytorch.org/data/main/tutorial.html#using-datapipes" rel="noopener nofollow ugc">an example</a> for Iterable-style (with <code>__iter__</code> method implemented), but it works the same way for Map-style (with custom <code>__getitem__</code> method).</p>