Issues with reproducibility

kirk86: <p>Hi folks, I’m trying to understand some aspects of reproducibility and it seems that I might have not understood something properly.</p>
<p>I have a toy multilayer perceptron  and every time I train and test it I get different results regarding test accuracy.</p>
<p>I’ve seeded all possible sources of randomness I could think of:</p>
<pre><code class="lang-auto">seed = 2021
os.environ['PYTHONHASHSEED'] = '2021'
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.backends.cudnn.benchmarks = False
torch.backends.cudnn.deterministic = True
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.set_deterministic(True)
</code></pre>
<p>But I still observe this discrepancy in accuracy every time a re-train and test the model.</p>
<p>I suspect that has to do with the fact that every time the model is constructed the weights are randomly initialised, hence, the different results in accuracy.</p>
<p>It seems like none of the above seeds fixes the weights of network for reproducibility, am I understanding this wrong or did I miss sth crucial regarding reproducibility. (And yes I’ve read the docs about reproducibility)</p>

ptrblck: <p>Seeing the code should use the same “random” values to initialize the model.<br>
If you think the model parameters are still different, you could save the <code>state_dict</code> of two different runs and compare the values.</p>
<p>In case the parameters are indeed equal, you could then add debug print statements and try to narrow down which iteration produces different results.</p>

kirk86: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
from a simple experiment with <code>numpy</code> and seeds it looks that no matter whether you set the seed or not consequent call to <code>numpy.ranodm.randn</code> will yield different results.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e_2_690x237.png" alt="image" data-base62-sha1="zULJwyVJRJ8hBacpc6dLSjkRmAS" width="690" height="237" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e_2_690x237.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e_2_1035x355.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbb6e9b74de8fdc586d1354211b1c80a7f197a7e_2_1380x474.png 2x" data-dominant-color="062C35"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2026×698 67.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In that case why do we expect whenever we create an instance of the model multiple times (e.g. <code>modelA = MLP(), modelB = MLP()</code>) to have same weights? I would expect that <code>modeA</code> and <code>modelB</code> to have different weights due to random initialisation, no?</p>

ptrblck: <aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="112552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>In that case why do we expect whenever we create an instance of the model multiple times (e.g. <code>modelA = MLP(), modelB = MLP()</code>) to have same weights?</p>
</blockquote>
</aside>
<p>We don’t and it’s expected that sequential calls to the pseudo random number generator will yield different results. Otherwise you would be sampling constant values, which is wrong.</p>
<p>Seeding the code makes sure that the sequence of random numbers are equal between runs.</p>
<pre data-code-wrap="python"><code class="lang-python">np.random.seed(2809)
print(np.random.randn(10))
&gt; [ 0.23747478  0.56083014 -1.1650379   0.12482746  0.04529476 -0.60137682
 -0.18559365  0.57410855 -0.53776209 -0.03972935]

print(np.random.randn(10))
&gt; [-0.85086083  0.25095493  0.30150578  1.15011173  0.36414967  0.3197194
 -1.60004496 -0.74648366  0.65739336  0.15710605]

np.random.seed(2809)
print(np.random.randn(10))
&gt; [ 0.23747478  0.56083014 -1.1650379   0.12482746  0.04529476 -0.60137682
 -0.18559365  0.57410855 -0.53776209 -0.03972935]

print(np.random.randn(10))
&gt; [-0.85086083  0.25095493  0.30150578  1.15011173  0.36414967  0.3197194
 -1.60004496 -0.74648366  0.65739336  0.15710605]
</code></pre>
<p>The <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">Wikipedia article</a> explains it in more detail.</p>

kirk86: <p>Thanks for the example, I’m not sure how you achieved random numbers to be equal between runs. For instance, in the example below they all seem different to me, no?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import random
&gt;&gt;&gt; os.environ['PYTHONHASHSEED'] = '2021'
&gt;&gt;&gt; random.seed(2021)
&gt;&gt;&gt; np.random.seed(2021)
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 1.48860905,  0.67601087, -0.41845137],
       [-0.80652081,  0.55587583, -0.70550429],
       [ 1.13085826,  0.64500184,  0.10641374]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 0.42215483,  0.12420684, -0.83795346],
       [ 0.4090157 ,  0.10275122, -1.90772239],
       [ 1.1002243 , -1.40232506, -0.22508127]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[-1.33620597,  0.30372151, -0.72015884],
       [ 2.5449146 ,  1.31729112,  0.0726303 ],
       [-0.25610814,  0.13801041,  1.14723599]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 1.37626076, -0.47218397,  0.5240849 ],
       [ 1.48510793,  1.48243225,  0.72813051],
       [-0.38964808,  0.27889376,  0.0519002 ]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[-1.04474368, -0.16150753, -2.79353057],
       [ 0.36164801,  0.24010758,  0.47781228],
       [ 0.20885194,  0.91791163, -1.41177784]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 1.22423572, -0.54565772,  0.90674805],
       [-0.98261724, -0.63316189,  0.82552024],
       [-1.28449686, -0.34730878, -1.07776075]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 0.97257495,  0.41219116, -0.13208604],
       [-0.67335092,  1.22222171, -0.92641342],
       [ 1.4249938 , -0.12478993,  0.70549192]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[ 0.7195475 ,  0.14146394, -0.65444848],
       [-0.67210417,  0.87036849,  1.00324527],
       [-0.36644983,  1.12805168,  0.7925838 ]])
&gt;&gt;&gt; np.random.randn(3,3)
array([[-1.75084239, -0.80886945, -1.41260181],
       [-0.8030097 ,  0.79828047,  1.75599622],
       [-0.0755042 ,  0.42017682,  0.15655606]])
</code></pre>

vaibhavballoli: <p>Terminal 1:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; np.random.seed(2809)
&gt;&gt;&gt; 
&gt;&gt;&gt; print(np.random.randn(10))
[ 0.23747478  0.56083014 -1.1650379   0.12482746  0.04529476 -0.60137682
 -0.18559365  0.57410855 -0.53776209 -0.03972935]
&gt;&gt;&gt;
</code></pre>
<p>Terminal 2:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; np.random.seed(2809)
&gt;&gt;&gt; 
&gt;&gt;&gt; print(np.random.randn(10))
[ 0.23747478  0.56083014 -1.1650379   0.12482746  0.04529476 -0.60137682
 -0.18559365  0.57410855 -0.53776209 -0.03972935]
&gt;&gt;&gt;
</code></pre>
<p>In <a class="mention" href="/u/ptrblck">@ptrblck</a> 's example, the <code>np.random.seed</code> method is called again which “resets” the generator and the same numbers are generated. The random numbers generated will be different every consecutive time it’s called once the seed is set, but the same sequence of numbers will be generated if you initialize it from the same seed from different instances.</p>

kirk86: <p>I see, thanks for the clarification I missed the 2nd call to <code>np.random.seed()</code> in <a class="mention" href="/u/ptrblck">@ptrblck</a> example.</p>
<p>I suppose that works fine with scripts running on terminal, but maybe its problematic with interactive notebooks, I guess calling the <code>seed()</code> function from cell to cell will be quite annoying?</p>

ptrblck: <aside class="quote no-group" data-username="kirk86" data-post="7" data-topic="112552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>I guess calling the <code>seed()</code> function from cell to cell will be quite annoying?</p>
</blockquote>
</aside>
<p>Why would you want to do that? Resetting the seed would break the assumption that you are “randomly” initializing the model, data etc.<br>
If you want to reuse the same parameters, I would rather load the <code>state_dict</code> of another model instead of relying in the seed.</p>