Disregard some images from the dataloader

mzimmer: <p>Suppose I have a train dataloader that yields batches of images for training. After some time of training I want to discard a certain group of images and I want the dataloader to only provide me with the images that I consider worth training on.</p>
<p>How can I realize this with a Sampler or Loader? It would be good to dynamically change the set of relevant images efficiently.</p>
<p>Thanks a lot!</p>

tom: <p>A very quick way could be to use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedRandomSample</a> with very small / 0 weight for the bits you don’t want.<br>
Note that things might break if you change the epoch length and tools expect the epochs to be of fixed length (<code>len(dl)</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

mzimmer: <p>Thanks for the quick answer. WeightedRandomSampler sounds promising, however I plan on changing the set of used samples quite often.</p>
<p>So I think it makes more sense to set some kind of boolean tensor within the dataloader and then modify the dataloader instead of the sampler.</p>

tom: <p>Last I looked, the dataloader would call the sampler at the beginning of an epoch. Given that instantiating a dataloader isn’t that much more expensive than starting an epoch (i.e. instantiating the iterator), I’m not sure that what you have in mind is easy to implement or much more efficient.</p>
<p>The obvious other solution is to forego the dataloading and produce a dataset (possibly an iterative dataset) that accomplishes this (in a well-optimized pipeline having multiple workers might not be as crucial as it seems to be initially).</p>
<p>That said, while I do believe that using the sampler is likely a good idea, it’s totally possible that you’ll have a much better solution matching your workflow. And if that’s the case, I’d be really keen to hear how you did it.</p>
<p>Best regards</p>
<p>Thomas</p>

mzimmer: <p>Dear Thomas, thanks again for the quick reply, really appreciate it.</p>
<p>In the meanwhile, I found <a href="https://discuss.pytorch.org/t/solved-will-change-in-dataset-be-reflected-on-dataloader-automatically/10206">this post</a>. Apparently I can simply change the dataset.</p>
<p>The torchvision datasets seem to contain the attribute <code>self.data</code> which I will try to backup in a different variable and overwrite by a subset of the original data. Hope this will lead to an efficient training on a smaller training set.</p>
<p>Will get back to you once I see how this goes, in case you have any concerns please let me know!<br>
Best,<br>
Max</p>

tom: <p>Yeah, changing the dataset is always a good if a bit hacky method to fix things.<br>
One thing to keep in mind is that it won’t propagate to the workers of a dataloader until they are restarted.</p>

mzimmer: <p>Not sure if this is true though, I see two posts claiming that this will work even with multiple workers, see <a href="https://discuss.pytorch.org/t/dynamic-dataloaders-for-on-the-fly-modifications/78870/12">this answer</a> or <a href="https://discuss.pytorch.org/t/solved-will-change-in-dataset-be-reflected-on-dataloader-automatically/10206/4">this one</a> for example.</p>
<p>In any case, will verify this! Thanks for the advice</p>

ptrblck: <p>Note that the linked post mention the <code>Dataset</code> manipulation has to be done after each epoch and without using persistent workers, so <a class="mention" href="/u/tom">@tom</a>’s warning fits into this.</p>

mzimmer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yes I noticed that, thanks for pointing out again.</p>
<p><a class="mention" href="/u/tom">@tom</a> In case you’re still interested, I basically just used <a href="https://discuss.pytorch.org/t/solved-will-change-in-dataset-be-reflected-on-dataloader-automatically/10206/2">this solution</a> and modified it as such that I simply create a backup copy of the data and target tensors to which I can modify and to which I can revert at any time.</p>
<p>Thanks for your help!</p>