How workers work under the hood?

ado_sar: <p>I am trying to understand how workers are initialized, what data is copied/shared and in what order they are called.</p>
<p>I have the following snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.utils.data import DataLoader, Dataset


def worker_init_fn(worker_id):
    info = torch.utils.data.get_worker_info()
    print(id(info.dataset), worker_id)


class DummyDataset(Dataset):
    def __init__(self):
        self.samples = list(range(8))

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        print(f'Worker {torch.utils.data.get_worker_info().id} is used')
        return self.samples[idx]


ds = DummyDataset()
train_loader = DataLoader(
        ds, num_workers=4,
        worker_init_fn=worker_init_fn, batch_size=2,
        )

for i, data in enumerate(train_loader):
    pass

</code></pre>
<p>which outputs:</p>
<pre data-code-wrap="bash"><code class="lang-bash">140276885599920 0                                                                                                                                                                             
140276885599920 1                                                                                                                                                                             
Worker 0 is used                                                                                                                                                                              
Worker 0 is used                                                                                                                                                                              
140276885599920 2                                                                                                                                                                             
Worker 2 is used                                                                                                                                                                              
Worker 2 is used                                                                                                                                                                              
Worker 1 is used                                                                                                                                                                              
Worker 1 is used                                                                                                                                                                              
140276885599920 3                                                                                                                                                                             
Worker 3 is used                                                                                                                                                                              
Worker 3 is used 
</code></pre>
<ul>
<li>Workers are not initialized simultaneously. As it can be seen, before initializing worker 2, worker 0 is already used. Is this correct?</li>
<li>Do they make a shallow or deep copy the dataset? From the output, it seems that each worker is passed the same dataset. However, I am not sure, since it can be the case that each previously dataset is garbage collected and the new one just happens to have the same <code>id</code> as the previous one.</li>
</ul>

ptrblck: <p>If you initialize the <code>self.samples</code> as a <code>tensor</code> and manipulate it inplace via:</p>
<pre data-code-wrap="python"><code class="lang-python">    def __getitem__(self, idx):
        print(f'Worker {torch.utils.data.get_worker_info().id} is used')
        self.samples[:] += 0.1
        return self.samples[idx]
</code></pre>
<p>you’ll see that each value is bumped twice only, indicating each worker uses its own <code>dataset</code> copy. The original <code>ds.samples</code> tensor is also keeping its originally assigned values after the <code>DataLoader</code> loop.</p>

ado_sar: <p>Thanks for the answer <a class="mention" href="/u/ptrblck">@ptrblck</a>! I was confused because I didn’t understand how multiprocessing works under the hood. At least in Unix, copy-on-write is enabled, meaning that the datasets passed to the workers can’t affect each other (and also the original dataset which runs under the <code>main</code> process). I have also written an <a href="https://stackoverflow.com/a/79261675/16568990" rel="noopener nofollow ugc">answer in StackOverflow</a> to better express these ideas. It would be very helpful if you could comment on the SO question, because I think the answers there are wrong and can potentially lead to misunderstandings for future readers and users of PyTorch.</p>