PyTorch VERY different results on different machines using docker and CPU

alexus37: <p>I am trying to create some CI tests for a deep learning model and therefore created a Dockerfile to install PyTorch and all my requirements and finally run the tests.</p>
<pre><code class="lang-auto">FROM pytorch/pytorch
ADD . / project/
RUN (cd project/; pip install -r requirements.txt)
CMD ( cd project/; pytest -v --cov=my_project)
</code></pre>
<p>The tests are basically computing an image from 0 - 1  and comparing it to a reference image (saved as npy). The test is checking<br>
if the avg L2 norm of the pixels is below a threshold of <code>1e-7</code>.</p>
<pre><code class="lang-python">diff_image = np.linalg.norm(target_image_np - reference_image_np, axis=2)
avg_error = np.mean(diff_image)
assert avg_error &lt; 1e-7
</code></pre>
<p>The tests pass 12/15 test cases. However, 3 cases are failing quite badly.</p>
<pre><code class="lang-auto">=========================== short test summary info ============================
FAILED test_nst.py::test_nst_gatys - assert 0.0021541715 &lt; 1e-07       
FAILED test_nst.py::test_nst_gatys_style - assert 0.12900369 &lt; 1e-07
FAILED test_nst.py::test_nst_wct - assert 0.027357593 &lt; 1e-07
=================== 3 failed, 12 passed in 670.27s (0:11:10) ===================
</code></pre>
<p>The weird thing is that this ONLY happens on the CI server. On my local machine, all tests pass. Does anybody have an idea why this is happening? As far as I know, using the CPU as well as fixed seeds should return at least results which differ only numerically.</p>
<p>Thanks for any feedback!</p>

jcowles: <p>We are experiencing a very similar issue:</p>
<aside class="quote quote-modified" data-post="1" data-topic="98608">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jkyl/48/29550_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/inference-non-deterministic-on-windows-cpu/98608">Inference non-deterministic on Windows CPU</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hello, 
I have observed non-determinism on the Windows CPU build of torch==1.6.0, related to fully-convolutional models’ ability to “tile”. 
By “tiling”, I am referring to the property of <a href="https://datascience.stackexchange.com/a/16084" rel="noopener nofollow ugc">translation equivariance</a>, whereby translating the input to a function results in an output that is translated by the same amount. This of course is the basic logic behind convolution, and it can be straightforwardly generalized to models which consist only of convolutional layers (including transpose conv, near…
  </blockquote>
</aside>


alexus37: <p>Thanks for your post I was not setting the number of threads. This fixed the issue for me. My final code looks like that:</p>
<pre><code class="lang-python">    np.random.seed(42)
    torch.manual_seed(42)
    os.environ["PYTHONHASHSEED"] = "42"
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.set_num_threads(1)
</code></pre>