Output from network varies depending on CPU or GPU

abailey: <p>Hi,</p>
<p>I have come across a problem where despite setting random seeds, I obtain different outputs from a simple network depending on whether I use CPU or GPU. I also receive different CPU results using different computers but receive the same GPU results.</p>
<p>The model weights are equal as is the randomly generated input “a”, however differences occur after passing “a” through the network. I also get the same results using model.eval() or model.train().</p>
<p>So my 2 questions are:</p>
<ol>
<li>Why are the results different when using CPU vs GPU even though random seeds have been set?</li>
<li>Probably linked to the first but why are the CPU results across computers different?</li>
</ol>
<p>My results using:<br>
System details: intel i5-7500, GTX 1060 6GB, 32GB RAM, Samsung Evo 850 500GB<br>
nvidia-smi: 435.21</p>
<pre><code class="lang-auto">output[0, :]: [[-0.18105512857437134], [-0.9114810228347778], [-0.5673332810401917], [-1.0145820379257202]]
output[0, :]: [[-0.18105511367321014], [-0.9114810228347778], [-0.5673332214355469], [-1.0145819187164307]]
</code></pre>
<p>When I run this code on another computer using:<br>
Intel® Xeon® Gold 6134 CPU @ 3.20GHz, GTX 1080 Ti<br>
nvidia-smi: 430.50</p>
<pre><code class="lang-auto">output[0, :]: [[-0.18105512857437134], [-0.9114810228347778], [-0.5673332810401917], [-1.0145820379257202]]
output[0, :]: [[-0.18105512857437134], [-0.9114810228347778], [-0.5673332214355469], [-1.0145819187164307]]

</code></pre>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy
import random
import math


class SimpleModel1(nn.Module):
    def __init__(self):
        super(SimpleModel1, self).__init__()
        self.hidden = nn.Linear(in_features=6,
                                out_features=1)
        self.init_weights()

    def init_weights(self):
        self.init_layer(self.hidden)

    def init_layer(self, layer):
        (n_out, n) = layer.weight.size()
        std = math.sqrt(2. / n)
        scale = std * math.sqrt(3.)
        layer.weight.data.uniform_(-scale, scale)

        if layer.bias is not None:
            layer.bias.data.fill_(0.)

    def forward(self, x):
        x = self.hidden(x)
        return x


if __name__ == '__main__':
    model_weights = []
    pre_a = []
    post_a = []
    for i in range(2):
        chosen_seed = 0
        torch.manual_seed(chosen_seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        torch.cuda.manual_seed_all(chosen_seed)
        numpy.random.seed(chosen_seed)
        random.seed(chosen_seed)
       
        # (batch, time_frame, feature)
        a = torch.rand(2, 4, 6)
        model = SimpleModel1()

        if i == 0:
            model.cuda()
            a = a.cuda()
        test_model1 = []
        test_a_1 = a
        for params in model.parameters():
            test_model1.append(params.data)

        model.train()

        output = model(a)
        test_model2 = []
        for params in model.parameters():
            test_model2.append(params.data)
        print(f"output[0, :]: {output[0, :, :].tolist()}")
        model_weights.append([test_model1[0], test_model2[0]])
        pre_a.append(test_a_1)
        post_a.append(output)

    for i in range(2):
        if torch.all(torch.eq(model_weights[0][i].cpu(), model_weights[1][i])):
            print('Model Weights OK')
        else:
            print('Error in Weights')
    if torch.all(torch.eq(pre_a[0].cpu(), pre_a[1])):
        print('Pre-model inputs OK')
    else:
        print('Error in pre_model inputs')
    if torch.all(torch.eq(post_a[0].cpu(), post_a[1])):
        print('Post-model inputs OK')
    else:
        print('Error in post_model inputs')

</code></pre>
<p>My environment details:</p>
<pre><code class="lang-auto">name: repro
channels:
  - pytorch
  - anaconda
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - blas=1.0=mkl
  - ca-certificates=2019.11.27=0
  - certifi=2019.11.28=py37_0
  - cudatoolkit=10.1.243=h6bb024c_0
  - freetype=2.9.1=h8a8886c_1
  - intel-openmp=2019.5=281
  - jpeg=9b=h024ee3a_2
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_0
  - mkl=2019.5=281
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py37hfd86e86_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_1
  - olefile=0.46=py37_0
  - openssl=1.1.1=h7b6447c_0
  - pillow=7.0.0=py37hb39fc2d_0
  - pip=19.3.1=py37_0
  - python=3.7.3=h0371630_0
  - pytorch=1.4.0=py3.7_cuda10.1.243_cudnn7.6.3_0
  - readline=7.0=h7b6447c_5
  - setuptools=44.0.0=py37_0
  - six=1.13.0=py37_0
  - sqlite=3.30.1=h7b6447c_0
  - tk=8.6.8=hbc83047_0
  - torchvision=0.5.0=py37_cu101
  - wheel=0.33.6=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0

</code></pre>

ptrblck: <p>The differences you are seeing are approx. <code>1e-6</code> which comes most likely due to the limited floating point precision of <code>float32</code>.<br>
The order of operations might yield different results as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, 10)
s1 = x.sum()
s2 = x.sum(0).sum(0).sum(0)
print((s1 - s2).abs().max())
&gt; tensor(3.8147e-06)
</code></pre>
<p>I assume the difference in CPU architecture between both posted CPUs might explain the difference, but that’s just by best guess.</p>
<p>Also, the implementation of the pseudo-random number generator might be different for different hardware devices (e.g. CPU vs. GPU).</p>

abailey: <p>Thanks for your answer, I tried increasing model complexity and printing out the maximum difference between GPU and CPU results and I think you are correct, the highest difference I received was 2.3543834686279297e-06.</p>
<p>Architecture difference is also a good guess.</p>