DataLoader workers generate the same random augmentations

Grzegorz_Los: <p>The question I’m about to ask is probably not PyTorch-specific, but I encountered it in context of PyTorch DataLoader.</p>
<p>How do you properly add random perturbations when data is loaded and augmented by several processes?</p>
<p>Let me show on a simple example that this is not a trivial question. I have two files:</p>
<ul>
<li>
<strong>augmentations.py</strong>:</li>
</ul>
<pre><code class="lang-auto">import numpy as np
import os

class RandomAugmentation:
    def __call__(self, obj):
        perturbation = np.random.randint(10)
        print(os.getpid(), perturbation)
        return obj
</code></pre>
<ul>
<li><strong>main.py</strong></li>
</ul>
<pre><code class="lang-auto">import numpy as np
from time import sleep

from torch.utils.data import Dataset, DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import Compose, ToTensor, Resize

from augmentations import RandomAugmentation

PATH = ...

transform = Compose([
    RandomAugmentation(),
    Resize((16, 16)),
    ToTensor()
])
ds = ImageFolder(PATH, transform=transform)
dl = DataLoader(ds, batch_size=2, num_workers=3)

for epoch_nr in range(2):
    for batch in dl:
        break
    sleep(1)
    print('-' * 80)
</code></pre>
<p>In the <strong>main.py</strong> we generate batches of data like we would in a regular image recognition task. <code>RandomAugmentation</code> prints output of RNG that would be used for data augmentation in a real task.</p>
<p>On my machine the output from running <strong>main.py</strong> was:</p>
<pre><code class="lang-auto">20909 6
20909 8
20908 6
20910 6
20908 8
20910 8
20909 7
20909 6
20908 7
20908 6
20910 7
20908 0
20910 6
20908 1
--------------------------------------------------------------------------------
20952 6
20953 6
20952 8
20952 7
20953 8
20952 6
20954 6
20952 0
20952 1
20953 7
20954 8
20953 6
20954 7
20954 6
--------------------------------------------------------------------------------
</code></pre>
<p>Clearly, i-th perturbation added by one worker is exactly the same as i-th perturbation added by any other worker (in this example 6, 8, 7, 6, and so on). What’s even worse, in the next epoch data loader is kind of reset, and generates exactly the same (perturbated) data as before, what completely ruins the entire idea of data augmentation.</p>
<p>For the time being I designed such a workaround:<br>
<strong>augmentations.py</strong></p>
<pre><code class="lang-auto">import numpy as np
from time import time
import os

RNG_PID = None
RNG = None


def get_rng():
    global RNG
    global RNG_PID
    if os.getpid() != RNG_PID:
        RNG = np.random.RandomState([int(time()), os.getpid()])
        RNG_PID = os.getpid()
        print("Initialize RNG", int(time()), os.getpid())
    return RNG

class RandomAugmentation:
    def __call__(self, obj):
        rng = get_rng()
        perturbation = rng.randint(10)
        print(os.getpid(), perturbation)
        return obj
</code></pre>
<p>However, it doesn’t seem like a very elegant solution to me.</p>
<p>It would be great to hear how you deal with this matter <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>It’s a known issue, since you are using other libraries (numpy in your case) to generate random numbers.<br>
Each worker will duplicate numpy’s PRNG, so that you’ll see the same numbers.<br>
It’s described in the <a href="https://pytorch.org/docs/stable/notes/faq.html#my-data-loader-workers-return-identical-random-numbers">FAQ</a> a bit better.</p>
<p>You could sample random numbers using <code>torch.randint</code> or use <code>worker_init_fn</code> to set the seed for each numpy process.</p>

mikolchon: <p>I think it would be good to clarify this in the official tutorial <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a> since it also uses numpy’s random module.</p>

ptrblck: <p>That might be indeed a good idea. Would you be interested in implementing it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

mikolchon: <p>Ok I just submitted PR <a href="https://github.com/pytorch/tutorials/pull/1121" rel="nofollow noopener">https://github.com/pytorch/tutorials/pull/1121</a> with the changes. I just added a note after the Transforms example clarifying the potential issue.</p>