PyTorch result changes on the exact same setting if run sequentially

ilhamfp: <h2>The Problem</h2>
<p>Even after setting all random factors with seeds, PyTorch result changes on the exact same setting if run sequentially.</p>
<h2>To Reproduce</h2>
<p>I have created 2 Kaggle Kernel that trains &amp; test PyTorch on the same MNIST dataset five times. The kernels showcase the difference between setting random factors only once vs. each sequential run.</p>
<ol>
<li>
<p><a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29965892#Train" rel="nofollow noopener">Setting random factor once</a></p>
</li>
<li>
<p><a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29965960#Train" rel="nofollow noopener">Setting random factor on each sequential run</a></p>
</li>
</ol>
<h2>Expected behavior</h2>
<p>I expect running the same train &amp; test five times on <a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29965892#Train" rel="nofollow noopener">kernel #1</a> would produce the exact same result each time. But somehow, something changes on each iteration. I can’t pinpoint what changes on each iteration. Only by resetting the random factor on each iteration (<a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29965960#Train" rel="nofollow noopener">kernel #2</a>) I could achieve the exact same result each time.</p>
<h2>Environment</h2>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: version 3.7.2

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] msgpack-numpy==0.4.4.3
[pip] numpy==1.18.1
[pip] numpydoc==0.9.2
[pip] pytorch-ignite==0.3.0
[pip] pytorch-pretrained-bert==0.6.2
[pip] pytorch-transformers==1.1.0
[pip] torch==1.4.0
[pip] torchaudio==0.4.0a0+719bcc7
[pip] torchtext==0.5.0
[pip] torchvision==0.5.0
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2019.3                      199  
[conda] mkl-service               2.0.2            py36h7b6447c_0  
[conda] mkl_fft                   1.0.12           py36ha843d7b_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] pytorch                   1.4.0               py3.6_cpu_0  [cpuonly]  pytorch
[conda] pytorch-ignite            0.3.0                    pypi_0    pypi
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
[conda] pytorch-transformers      1.1.0                    pypi_0    pypi
[conda] torchaudio                0.4.0                      py36    pytorch
[conda] torchtext                 0.5.0                    pypi_0    pypi
[conda] torchvision               0.5.0                  py36_cpu  [cpuonly]  pytorch
</code></pre>

albanD: <p>Hi,</p>
<p>I am reading the code correctly that the difference is between:</p>
<pre><code class="lang-auto">for counter in range(1, 6):
    set_seed(seed=RANDOM_SEED)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">set_seed(seed=RANDOM_SEED)
for counter in range(1, 6):
</code></pre>
<p>?</p>
<p>If so, then it is expected as in the second case, when you recreate the net inside the for-loop, since the seed is not set, you get a different initialiazation for your net and so you will get different results.</p>

ilhamfp: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thank you for your reply.<br>
Yes, that’s the only difference.</p>
<p>Hmm, that’s really interesting. After reading your reply, I tried to print the 1st conv2d weight at each iteration while only setting seed once. You’re right that the initialization weight changed! Hence the results vary. <a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29976118#Train" rel="nofollow noopener">Link to kernel</a></p>
<p>But, I tried another experiment. This time I set the initialization weight to a fixed value. To my surprise, even though each iteration started from the same weight initialization, the results still change on each iteration. <a href="https://www.kaggle.com/ilhamfp31/pytorch-sequential-deterministic-test?scriptVersionId=29976513#Train" rel="nofollow noopener">Link to kernel</a></p>
<p>So what’s the real culprit?</p>

albanD: <p>I would guess that the order you get the samples in the dataloader will change as well?<br>
Anything that uses randomness will lead to differences if you don’t seed it.</p>

ilhamfp: <p>Hmm, but I do set <code>shuffle=False</code> on the DataLoader for train data.</p>

albanD: <p>I am not sure in your particular code.</p>
<p>Taking a step back. You should not rely on getting the exact same results as depending on the hardware and version you have, the results might change.<br>
This is due to floating points being non-exact and the fact that gradient descent has the tendency to increase any small difference very quickly. Leading to completely different results.<br>
For stable models, you will get similar final loss and accuracy. But you won’t get the exact same solution.</p>