Setting seeds does not give reproducible code

shrbrh: <p>Hi, I am trying to reproduce the results of my pytorch code, but it’s giving different values every time I run it. I have set the seed as follows:</p>
<pre><code class="lang-auto">seed = 42
numpy.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
tf.random.set_seed(seed)
tf.experimental.numpy.random.seed(seed)
tf.compat.v1.set_random_seed(seed)

# When running on the CuDNN backend, two further options must be set
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = True

#For tensorflow
os.environ['TF_CUDNN_DETERMINISTIC'] = '1'
os.environ['TF_DETERMINISTIC_OPS'] = '1'
	
# Set a fixed value for the hash seed
os.environ["PYTHONHASHSEED"] = str(seed)
</code></pre>
<p>Did I miss anything?</p>

eqy: <p>Have you narrowed it down to a particular operation or operation(s) that exhibit non-determinism? Note that the docs also recommend <code>torch.backends.cudnn.benchmark = False</code> for determinism: <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.13 documentation</a></p>

shrbrh: <p><a class="mention" href="/u/eqy">@eqy</a> I tried setting <code>torch.backends.cudnn.benchmark = False</code> but it’s still not working.<br>
I am actually training a GAN, and the loss values and discriminator probability outputs for real/fake are different every time I run the code.</p>

ptrblck: <aside class="quote no-group" data-username="shrbrh" data-post="3" data-topic="172792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b5e925/48.png" class="avatar"> shrbrh:</div>
<blockquote>
<p>probability outputs for real/fake are different every time I run the code.</p>
</blockquote>
</aside>
<p>Could you try to narrow down the part of the code which creates the non-deterministic values and post the code snippet here, please?</p>

shrbrh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Actually I am getting non-deterministic values for multiple outputs. Upon searching for solutions, I got a post <a href="https://discuss.pytorch.org/t/is-there-a-way-to-fix-the-random-seed-of-every-workers-in-dataloader/21687/5">here</a> that says setting <code>num_workers = 0</code>  for  <code>Dataloader</code> works. I tried that but it still doesn’t work for me. But by making a few changes I am thinking it has something to do with the <code>Dataloader</code> part of my code. Here is the snippet:</p>
<pre><code class="lang-auto">content_iter = iter(data.DataLoader(
    content_dataset, batch_size=4,
    sampler=InfiniteSamplerWrapper(content_dataset),
    num_workers=16))
style_iter = iter(data.DataLoader(
    style_dataset, batch_size=number_of_styles,
    sampler=InfiniteSamplerWrapper(style_dataset),
    num_workers=16))
</code></pre>
<p>Here,  <code>number_of_styles</code> is 19.<br>
And the <code>InfiniteSamplerWrapper </code> method is defined in a different file as follows:</p>
<pre><code class="lang-auto">import numpy as np
from torch.utils import data

def InfiniteSampler(n):
    # i = 0
    i = n - 1
    order = np.random.permutation(n)
    while True:
        yield order[i]
        i += 1
        if i &gt;= n:
            np.random.seed(1)
            order = np.random.permutation(n)
            i = 0


class InfiniteSamplerWrapper(data.sampler.Sampler):
    def __init__(self, data_source):
        self.num_samples = len(data_source)
        #print("Num samples:",self.num_samples)

    def __iter__(self):
        return iter(InfiniteSampler(self.num_samples))

    def __len__(self):
        return 2 ** 31
</code></pre>
<p>If I leave  <code>np.random.seed()</code> as it is without specifying any number inside the braces, I get all different values from the beginning. Whereas, if I use  <code>np.random.seed(1)</code> I get the exact values for the first iteration, then it starts to change from the second iteration.</p>

ptrblck: <p>Your code is unfortunately not executable so that I won’t be able to debug it. If you are using multiple worked you would have to use the <code>worker_init_fn</code> to seed third party libs such as numpy.</p>

nwn: <p><strong>Just a wild guess</strong>, therefor no guarantee that it will work. Have you tried setting the dataloader seed?</p>