Inconsistent evaluation between CPU and GPU

username: frod
cooked: <p>Hi.</p>
<p>Me and another colleague have been struggling on this problem for the last days and we are losing our sanity. Unfortunately the conditions to recreate this problems are so rare and fragile that I have to attach a ~600Mb data file to this post for people interested to reproduce this. This data file does not contain any personal or sensible data, it’s just some random Monte Carlo numbers generated by another program.</p>
<p>Anyway here is the problem:</p>
<p>We have a really simple model: 7 input neurons, going to 5 and then a single output. Everything is made with linear layer and with sigmoid activations. In the example below, the model doesn’t even need to be trained for this problem to appear.</p>
<p>When we evaluate this model on some large datasets (~10 Million points), there are some inconsistencies between the CPU and GPU evaluations. In particular, the CPU is always right but the GPU starts giving some random results after about 2 Million points.</p>
<p>Here are some plots that show this behavior:</p>
<ul>
<li>
<p>histogram of the difference between cpu evaluations and gpu evaluations. You can see a peak at 0 where the two evaluations coincide</p>
</li>
<li>
<p>plot of the same difference. You can see that the two evaluations coincide for the first 2 Million points and then it becomes noise.</p>
</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/865b300d312ed44bc4ef3b524011562c996bdc65.png" data-download-href="https://discuss.pytorch.org/uploads/default/865b300d312ed44bc4ef3b524011562c996bdc65" title="plots"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/865b300d312ed44bc4ef3b524011562c996bdc65.png" alt="plots" data-base62-sha1="jazlGAyhmJfusmceYMtRRm6G2ON" width="406" height="500" data-dominant-color="D8E6EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plots</span><span class="informations">410×504 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you want to reproduce these plots here is a minimal working code:</p>
<pre><code class="lang-auto">import torch, os, h5py
from torch import nn
import matplotlib.pyplot as plt

class Model(nn.Module):    
    def __init__(self, input_dim, output_dim, hidden_units):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_units, bias = False)
        self.fc2 = nn.Linear(hidden_units, output_dim, bias = False)

    def forward(self, x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        return x

#create the model
model = Model(7, 1, 5)

#read the data file
data = h5py.File(os.getcwd() + '/data.h5', 'r')
test_input = torch.Tensor((data['Data'])[()])

#shuffle the data
test_input = test_input[torch.randperm(test_input.size(0))]

#get indices for which variable 6 is between 350 and 450
idx1 = test_input[:, 6] &gt; 350
idx2 = test_input[:, 6] &lt; 450

#normalize data
norm_test_input = (test_input - test_input.mean(0))/test_input.std(0)

#take only data between 350 and 450
test_input_cut = norm_test_input[idx1 * idx2]

print(test_input_cut.shape)

# --- #

#evaluate the model on CPU
print("CPU")
with torch.no_grad():
    print(model(test_input_cut)[-10:])
    print(model(test_input_cut[-10:]))

    cpu_results = model(test_input_cut)

#evaluate the model on GPU
print("GPU")
model.cuda()    
test_input_cut = test_input_cut.cuda()
with torch.no_grad():
    print(model(test_input_cut)[-10:])
    print(model(test_input_cut[-10:]))

    gpu_results = model(test_input_cut)
                     
#plot difference between the two evaluations       
diff = (cpu_results - gpu_results.cpu()).squeeze().numpy()
plt.hist(diff, bins = 30)
plt.show()

plx = range(cpu_results.size(0))
ply = (cpu_results.squeeze() - gpu_results.squeeze().cpu()).numpy()
plt.plot(plx, ply)
plt.show()
</code></pre>
<p>And <a href="https://www.dropbox.com/s/cwsf9oz69pcqqqf/data.h5?dl=0" rel="noopener nofollow ugc">here is the 600MB .h5 file</a> (that needs to be put in the same folder of the python code) for which this problem appears.</p>
<p>This behavior is really fragile in this example: changing the two numbers (350 and 450) even slightly erases makes everything work fine. For different (or random generated) data we couldn’t find any “cut” that would make this appear, but from what we tried it doesn’t look like there are any problems with the input data. In the real case, i.e. in the full project code, this seems to happen more consistently. We can’t post the full source code, so this tiny fragile example is the only thing that we could come up with to reproduce the issue in a consistent way to be posted in a forum.</p>
<p>Thanks everybody in advance for your help.</p>

username: ptrblck
cooked: <p>This seems to be an interesting problem, but unfortunately I couldn’t reproduce this issue.<br>
I used your code and the provided data, but the differences on my system seem to be alright.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a06e5943d586817b4186637de9c3c7f5859dbe15.png" alt="res1" data-base62-sha1="mTeOBFR8wMWQIfNRzafE7XT6vVH" width="403" height="265"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/95bee2bd6354ca7173c49ca48329e1aa4252ffc7.png" alt="res2" data-base62-sha1="lmI73yoNGZakAQkietBDZ2ADxfp" width="396" height="259"></p>
<p>Also, printing the unique values for <code>diff</code> gives:</p>
<pre data-code-wrap="python"><code class="lang-python">np.unique(diff, return_counts=True)
(array([-1.1920929e-07, -5.9604645e-08,  0.0000000e+00,  5.9604645e-08,
         1.1920929e-07], dtype=float32),
 array([    765,  441331, 3331571,  101474,       2]))
</code></pre>
<p>which is in the expected floating point precision range.</p>
<p>I’m using a 1080Ti, 418.56  driver and CUDA10.1.</p>

username: frod
cooked: <p>Hi thanks for your answer. We tested this on two different machines and we get the same result.</p>
<p>Machine 1:</p>
<p>GPU: GTX 1070<br>
Pytorch: 1.1.0<br>
Cuda: 9.0.36<br>
Driver: 430.14</p>
<p>Machine 2:</p>
<p>GPU: GTX 1050<br>
Pytorch: 1.0.1<br>
Cuda: 9.0<br>
Driver: 430.86</p>
<p>It’s weird that you don’t find the same. Could it be related to CUDA 9 vs 10?</p>

username: ptrblck
cooked: <p>I’m not sure, but it would be interesting to see, if updating to CUDA10 helps.<br>
Would that be possible or are you stuck to CUDA9 on your machines?</p>

username: frod
cooked: <p>I have updated the GTX1050 machine to pytorch 1.1.0 and CUDA 10 and now the problems seem not to be there, neither in the example or in the full code. I wonder what was the problem then.</p>

username: ptrblck
cooked: <p>I could reproduce this issue on my system with CUDA9.0.<br>
I also debugged a little bit and found that the GPU outputs a constant value after <code>2097152</code> input values, which is exactly <code>2**21</code>.</p>
<pre><code class="lang-python">print(gpu_results[2**21:].shape)
&gt; torch.Size([1777991, 1])
print(gpu_results[2**21:].unique())
&gt; tensor([0.5127], device='cuda:0', grad_fn=&lt;NotImplemented&gt;)
</code></pre>
<p>I’m not sure, what’s going on, but maybe <a class="mention" href="/u/ngimel">@ngimel</a> might know some limitation regarding this magic number and CUDA9.</p>

username: ngimel
cooked: <p>2**21 number makes me think that you are hitting <a href="https://github.com/pytorch/pytorch/pull/22034" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/22034</a>. The solution is to update to cuda 9.2</p>