Handling GPU/CPU compute differences

Simon_Watson: <p>Hi All,</p>
<p>I am porting a computational graph from tensorflow v1 to pytorch and have hit an issue with my float32 data.<br>
The illustrative example below sums up the issue. I get a result which is a vector of either 37059.996 or 37061.0 depending on the hardware used for calculation with a float32 tensor.</p>
<pre><code class="lang-python"># set up arrays in numpy:
A = np.repeat(np.array([[1.0], [33.0], [0.0],  [1089.0], [0.0], [0.0], [35937.0], [0.0], [0.0], [0.0],  [0.9991], [0.0]]),1000,axis =1).T.astype('float32')
B = torch.empty(12,1).fill_(1.).astype('float32')

# show the dot product of the arrays. 
print(np.dot(A,B))

# take the arrays to the GPU and do the same math
print(torch.mm(torch.tensor(A,dtype=torch.float32).to('cuda'),torch.tensor(B,dtype=torch.float32).to('cuda')).to('cpu').detach().numpy())

# show the difference from the two approaches:
print(np.dot(A,B)-torch.mm(torch.tensor(A,dtype=torch.float32).to('cuda'),torch.tensor(B,dtype=torch.float32).to('cuda')).to('cpu').detach().numpy())
</code></pre>
<p>I’m completely open to going through my code and switching to torch.float64. However, before I do, could you advise:</p>
<ul>
<li>is switching to float64 the best solution or is there a ‘magic fix’ that newer users such as myself might not be aware of?</li>
<li>does anyone have experience of the kind of performance impact the float32 to float64 change incurs?</li>
<li>is this inconsistency between GPU and CPU computation understood and is there any documentation on the error bounds a person might expect? (I get that the cuda representation of a number is not actually the same as a numpy representation and am happy to learn more and accommodate if the documentation is available. )</li>
</ul>
<p>Thanks and regards,</p>
<p>Simon</p>

ptrblck: <p>Based on the error of ~1e-5 you are most likely running into small errors caused by the limited floating point precision.</p>
<ol>
<li>
<p>It’s not a magic fix, but will give you more precision, thus reducing the error if you are using a wider <code>dtype</code>.</p>
</li>
<li>
<p>On GPUs you would expect to see poor performance using <code>float64</code>.</p>
</li>
<li>
<p>It’s not necessarily only visible between CPU and GPU calculations, but depends on the order of operations which could also change on the same device as seen e.g. here:</p>
</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
print((s1 - s2).abs())
# tensor(1.9073e-05)
</code></pre>
<aside class="quote no-group" data-username="Simon_Watson" data-post="1" data-topic="157580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simon_watson/48/23069_2.png" class="avatar"> Simon_Watson:</div>
<blockquote>
<p>I get that the cuda representation of a number is not actually the same as a numpy representation</p>
</blockquote>
</aside>
<p>That’s not the case as both as using the IEEE floating point standard (unless you are using TF32 on Ampere GPUs). Take a look at <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article</a> for more information.</p>