Dataloader issues with multiple workers generating samples on the fly

Haydnspass: <p>Hey,</p>
<p>since I can generate my training data, I basically have access to unlimited datasets and generate the samples using a torch Dataset and Dataloader on the fly.<br>
But as generating samples is (medium) expensive I want to reuse the generated data for a limited lifetime, let’s say 10 epoches until I would get overfitting effects.</p>
<p>The thing is, when I use single threaded dataloader (i.e. num_workers=0) everything seems to be fine. However as soon as I use multiple workers, they do not seem to have access to write to the dataset, so checking for existing samples fails all the time and new samples need to be generated.<br>
The reason why I like doing this with torch’s dataset / dataloader is that I do not have to care about parallel programming and all of my CPU cores are nicely utilised to generate the data.</p>
<p>Summarised I basically want to do the following:</p>
<ul>
<li>Init Dataset class and give it a lifetime of 10 epoches</li>
<li>for each epoch, for each batch: draw sample from dataset. <code>if sample is None</code>, generate a new one and save it as let’s say <code>self.sample[index] = function_which_generates_samples()</code> and return otherwise use existing sample</li>
<li>after each epoch: dataset.time_til_death -= 1, if dataset.time_til_death == 0: self.samples = [None] * pseudo_length</li>
</ul>
<p><em>If it’s helpful you can find a dummy example below (works for num_workers=0 but not for more)</em></p>
<pre><code class="lang-auto">class DummyDataset(Dataset):

    def __init__(self, lifetime=5, ds_size=8):

        super().__init__()
        self.lifetime = lifetime
        self.time_til_death = lifetime
        self.ds_size = ds_size

        self.samples = None
        self.gt = None
        self._drop_dataset()

    def step(self):
        self.time_til_death -= 1
        if self.time_til_death &lt;= 0:
            self._drop_dataset()
            self.time_til_death = self.lifetime

    def _drop_dataset(self):
        self.samples = [None] * self.__len__()
        self.gt = [None] * self.__len__()

        print("Dropped.")

    def _new_sample(self):
        return torch.rand(1), torch.tensor([1])

    def __len__(self):
        return self.ds_size

    def __getitem__(self, index):

        if self.samples[index] is not None:
            print("Old sample.")
            sample = self.samples[index]
            gt = self.gt[index]

        else:
            print("New sample.")
            sample, gt = self._new_sample()
            self.samples[index] = sample
            self.gt[index] = gt

        return sample, gt


if __name__ == '__main__':

    ds = DummyDataset(lifetime=2, ds_size=2)
    dl = DataLoader(ds, batch_size=2, shuffle=True, num_workers=4)

    for e in range(3):
        print(f"Epoch {e}")
        for i, (sample, gt) in enumerate(dl):
            print(f'Batch: {i} Sample: {sample} GT {gt}')
            # pass

        ds.step()
</code></pre>

Haydnspass: <p>Or to shorten the question: Do Dataloaders have ‘write’ access to variables when using multiple workers?</p>

ptrblck: <p>The <code>Dataset</code> is copied for each worker, if I’m not mistaken, which will lose all changes performed on these copies.<br>
Have a look at <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py" rel="nofollow noopener">this example</a> of shared arrays and let me know, if that would help in your use case.</p>

Haydnspass: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Works like a charm <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>