Weird precision problem with linear layer

username: XFeiF
cooked: <p>I’m using a linear layer to reduce the dimension of input tensors.<br>
I have two input tensors converted from numpy arrays, these two arrays:</p>
<ol>
<li>they are generated by a same function</li>
<li>‘a1.npy’ is generated lonely</li>
<li>‘b1.npy’ is generated with other data</li>
<li>a1 equals to b1 ( I tried a lot of methods to compare them)</li>
</ol>
<p>Then, I convert these two arrays to torch tensor and copy them to cuda.<br>
Finally, I pass them to a linear layer, and the results are different!<br>
The difference is not like machine precision error, as they are bigger.</p>
<p>When I try to reproduce this with Google Colab GPU T4, the two results become the same again… But this phenomenon actually happened on my local machine.</p>
<p>Can anyone explain this?</p>
<p>My environment:</p>
<ul>
<li>torch 1.10.2+cu113</li>
<li>ubuntu 20.0</li>
<li>GPU: rtx3090, rtx4090</li>
</ul>
<p>To reproduce:<br>
<a href="https://drive.google.com/drive/folders/1IL6TjjPEPfD-XvvGtsxV0-pnkLYoUTjP" rel="noopener nofollow ugc">download data</a></p>
<pre><code class="lang-auto">import numpy as np
import torch

a1 = np.load('a1.npy')
b1 = np.load('b1.npy')
t1 = torch.tensor(a1).cuda()
t2 = torch.tensor(b1).cuda()
print('raw data diff: ', (t1 != t2).sum())

mlp = torch.load('linear.ckpt')
mlp.eval()

r1 = mlp(t1)
r2 = mlp(t2)

print('result diff:', (r1 != r2).sum())

# output
# &gt;&gt;&gt; raw data diff: tensor(0, device='cuda:0')
# &gt;&gt;&gt; result diff: tensor(127583, device='cuda:0')
</code></pre>

username: KFrank
cooked: <p>Hi XFeiF!</p>
<aside class="quote no-group" data-username="XFeiF" data-post="1" data-topic="189807" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xfeif/48/49520_2.png" class="avatar"> XFeiF:</div>
<blockquote>
<ol start="2">
<li>‘a1.npy’ is generated lonely</li>
<li>‘b1.npy’ is generated with other data</li>
<li>a1 equals to b1 ( I tried a lot of methods to compare them)</li>
</ol>
</blockquote>
</aside>
<p>Perhaps because <code>a1</code> and <code>b1</code> are created somewhat differently, their elements<br>
are stored differently internally even though they are equal.</p>
<p>Consider (in pytorch, not numpy):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.1.0'
&gt;&gt;&gt; t1 = torch.tensor ([[1, 2, 3], [4, 5, 6]])
&gt;&gt;&gt; t2 = torch.tensor ([[1, 4], [2, 5], [3, 6]]).T
&gt;&gt;&gt; torch.equal (t1, t2)
True
&gt;&gt;&gt; t1.stride()
(3, 1)
&gt;&gt;&gt; t2.stride()
(1, 2)
</code></pre>
<p>The <code>stride()</code> is telling you that <code>t1</code> is stored in row-major order, while <code>t2</code> is<br>
in column-major order.</p>
<aside class="quote no-group">
<blockquote>
<p>Then, I convert these two arrays to torch tensor</p>
</blockquote>
</aside>
<p>For efficiency, when converting to <code>Tensor</code> (by using <code>from_numpy()</code>), pytorch<br>
wraps the original numpy data in a <code>Tensor</code>, so the underlying (difference in)<br>
storage order will be preserved.</p>
<aside class="quote no-group">
<blockquote>
<p>and copy them to cuda.</p>
</blockquote>
</aside>
<p>I assume, but haven’t checked, that the <code>.stride()</code> of a cpu tensor is preserved<br>
when you move it to the gpu.</p>
<p>If so, your two cuda tensors, even though equal, will have differing storage<br>
order.  This could well cause the gpu to perform operations in the <code>Linear</code>’s<br>
matrix multiplications in a different order, leading to round-off error.</p>
<p>Check the <code>.stride()</code> of your gpu tensors.  (Also check the <code>.data_offset()</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>Finally, I pass them to a linear layer, and the results are different!<br>
The difference is not like machine precision error, as they are bigger.</p>
</blockquote>
</aside>
<p>Are you sure?  How have you verified that the difference is not round-off<br>
error (that could be rather larger than machine precision if it accumulated<br>
in, say, a big matrix multiplication)?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">print('raw data diff: ', (t1 != t2).sum())
...
print('result diff:', (r1 != r2).sum())

# output
# &gt;&gt;&gt; raw data diff: tensor(0, device='cuda:0')
# &gt;&gt;&gt; result diff: tensor(127583, device='cuda:0')
</code></pre>
</blockquote>
</aside>
<p>In what you’ve posted, you’ve only shown that <code>r1</code> and <code>r2</code> have lots of<br>
elements that aren’t exactly equal, perhaps differing by only some small<br>
round-off error.</p>
<p>Try looking at <code>torch.allclose (r1, r2, atol = 1.e-3)</code> (or maybe even<br>
<code>atol = 1.e-2</code>) and <code>(r1 - r2).abs().max()</code>.</p>
<p>If this doesn’t sort things out, could you <strong>post</strong> a complete, full-self-contained,<br>
runnable script (presumably using both numpy and pytorch) that reproduces<br>
your issue?  Please use randomly-generated or hard-coded data produced<br>
inside the script itself.  Please also post the results you get from running the<br>
script.</p>
<p>Best.</p>
<p>K. Frank</p>

username: XFeiF
cooked: <p>Thank you so much for your prompt and helpful response <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20">.<br>
Your solution to my question worked perfectly and resolved the issue I was facing.</p>
<p>After I checked the <code>stride</code> of <code>t1(a1)</code> and <code>t2(b1)</code>, I found that they are stored differently internally! One is stored in row-major order and the other is stored in column-major order.</p>
<p>I used the NumPy functions <code>ascontiguousarray</code> or <code>asfortranarray</code> to switch the stride of the original NumPy arrays.<br>
Finally, the outputs of the <code>mlp</code> layer are now the same! <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_peeking_eye.png?v=12" title=":face_with_peeking_eye:" class="emoji" alt=":face_with_peeking_eye:" loading="lazy" width="20" height="20"></p>