Dataloader shows strange behavior for num_workers >0

username: Deep_Shankar_Pandey
cooked: <p>I am trying to use information from the outside functions to decide which data to return. Here, I have added a simplified code to demonstrate the problem. When I use <code>num_workers = 0</code>, I get the desired behavior (The output after 3 epochs is 18). But, when I increase the value of <code>num_workers</code>, the output after each epoch is the same. And the global variable remains unchanged.</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader

x = 6
def getx():
    global x
    x+=1
    print("x: ", x)
    return x

class MyDataset(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, index):
        global x
        x = getx()
        return x
    
    def __len__(self):
        return 3

dataset = MyDataset()
loader = DataLoader(
    dataset,
    num_workers=0,
    shuffle=False
)

for epoch in range(4):
    for idx, data in enumerate(loader):
        print('Epoch {}, idx {}, val: {}'.format(epoch, idx, data))

</code></pre>
<p>The final output when <code>num_workers=0</code> is 18 as expected. But when <code>num_workers&gt;0</code>, x remains unchanged (The final output is 6).</p>
<p>How can I get a similar behavior as <code>num_workers=0</code> using <code>num_workers&gt;0</code>(i.e.How to ensure the <code>__getitem__</code> function of dataloader changes the global variable <code>x</code>'s value )?</p>
<p>It would be great if someone can explain the problem and how to fix it.</p>

username: suraj.pt
cooked: <p>Good question!</p>
<p>From the <a href="https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading">docs</a>:</p>
<blockquote>
<p>In this mode, each time an iterator of a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader"> <code>DataLoader</code> </a> is created (e.g., when you call  <code>enumerate(dataloader)</code> ),  <code>num_workers</code>  worker processes are created. At this point, the  <code>dataset</code> ,  <code>collate_fn</code> , and  <code>worker_init_fn</code>  are passed to each worker, where they are used to initialize, and fetch data. This means that dataset access together with its internal IO, transforms (including  <code>collate_fn</code> ) runs <em>independently</em> in the worker process.</p>
</blockquote>
<p>This implies:</p>
<ul>
<li>
<p>when <code>num_workers=0</code>, for each epoch, data fetching is done by MAIN process</p>
</li>
<li>
<p>when <code>num_workers=1</code>, for each epoch, MAIN spawns a new worker each time <code>enumerate(loader)</code> is called. At each epoch, you should expect output of 7,8,9</p>
</li>
<li>
<p>when <code>num_workers=2</code>, for each epoch, MAIN spawns 2 new workers each time <code>enumerate(loader)</code> is called. Since <code>len(dataset)=3</code>, worker1 gets 2 indices, worker2 gets 1 index. You should expect output of 7,8 from worker1, and 7 from worker2.</p>
</li>
<li>
<p>If you change <code>len(dataset)</code> to 4,</p>
<ul>
<li>with <code>num_workers=2</code>, you should expect 7,8,7,8</li>
<li>with <code>num_workers=3</code>, you should expect 7,8,7,7</li>
</ul>
</li>
</ul>
<p>To bypass this GIL limitation, you can look into multiprocessing communication constructs.</p>

username: Deep_Shankar_Pandey
cooked: <p>For this problem, shared tensors can be used.</p>
<p>I asked the same question on stack overflow and the solution works.<br>
<a href="https://stackoverflow.com/questions/63460992/how-do-i-fix-the-dataset-to-return-desired-output-pytorch" rel="nofollow noopener">Stack overflow Answer</a></p>
<p>Hope this helps others facing similar problem.</p>