Discrepancy in matmul while batching

Vignesh_Baskaran: <p>I am trying to understand the discrepancy that happens while performing matrix multiplication in batch.<br>
To summarize I am trying to do the following:</p>
<ol>
<li>matmul(matrix, tensor) → Slice the output → sliced output</li>
<li>slice the input tensor →  matmul(matrix, sliced_tensor) → sliced output</li>
</ol>
<p>I expect the results to match exactly but that is not the case. In detail:</p>
<p>In the first step I  perform the  matmul between a tensor and a matrix and slice the  second item (in fact could be any item) of the output. Here is my code</p>
<pre><code class="lang-python">torch.manual_seed(0)
batch_size = 16
input_size = 64
seqlen = 50
hidden_size = 32

tensor = torch.randn(batch_size, input_size, seqlen)
matrix = torch.randn(hidden_size, input_size)
output = torch.matmul(matrix, tensor)
</code></pre>
<p>The output is of shape: <code>batch_size, hidden_dim, seqlen</code> as expected. I slice the second item of the output batch and print the result below:</p>
<pre><code class="lang-python">print(output[2])
tensor([[ -7.3307,   6.8155, -17.1523,  ...,   0.8284,  -8.5264,  -6.9085],
        [ -7.0253,   0.8953,  -3.8731,  ...,  -0.3406,  -0.8881,  14.2574],
        [  4.6008,  -2.4289,   2.7909,  ..., -13.3013,  -3.2574,  -5.8462],
        ...,
        [ -7.6053,  -3.6847,  -1.0073,  ...,  -5.0458, -16.5095,   6.8555],
        [  5.7580,   8.9886,  -3.3833,  ...,   6.3746,  -4.7638,  -0.6257],
        [  7.8220, -24.1010,   8.7975,  ...,   0.4481,  -4.6842, -19.7411]])
</code></pre>
<p>In the next step,  I slice the second item of the input batch and repeat the same operation and I expect the result to match with what I got above while performing the matrix multiplication in batch</p>
<pre><code class="lang-python">output_2 = torch.matmul(matrix, tensor[2])
</code></pre>
<p>The output of the above operation is:</p>
<pre><code class="lang-python">output_2
tensor([[ -7.3307,   6.8155, -17.1523,  ...,   0.8284,  -8.5264,  -6.9085],
        [ -7.0253,   0.8953,  -3.8731,  ...,  -0.3406,  -0.8881,  14.2574],
        [  4.6008,  -2.4289,   2.7909,  ..., -13.3013,  -3.2574,  -5.8462],
        ...,
        [ -7.6053,  -3.6847,  -1.0073,  ...,  -5.0458, -16.5095,   6.8555],
        [  5.7580,   8.9886,  -3.3833,  ...,   6.3746,  -4.7638,  -0.6257],
        [  7.8220, -24.1010,   8.7975,  ...,   0.4481,  -4.6842, -19.7411]])
</code></pre>
<p>Although the outputs look similar visually, there is a slight discrepancy between them :</p>
<pre><code class="lang-python">matching = ((output[2] == output_2).sum() / output_2.numel()).item()
print(f'Percentage of output matching: {matching}')
Percentage of output matching: 0.9662500023841858
</code></pre>
<p>I checked and found the result of <code>torch.allclose</code> is True. Superficially I understand that this is because of some floating point error or something like that. I have the following questions:</p>
<ol>
<li>Is this normal to have a slight discrepancy?</li>
<li>Can someone please explain why this discrepancy happens with some under the hood explanation or please point me to resources to understand this in depth?</li>
</ol>
<p>Thank you</p>

KFrank: <p>Hi Vignesh!</p>
<aside class="quote no-group" data-username="Vignesh_Baskaran" data-post="1" data-topic="125023" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vignesh_baskaran/48/39656_2.png" class="avatar"> Vignesh_Baskaran:</div>
<blockquote>
<ol>
<li>Is this normal to have a slight discrepancy?</li>
</ol>
</blockquote>
</aside>
<p>Yes, such a discrepancy is to be expected due to floating-point<br>
round-off error.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>Can someone please explain why this discrepancy happens with some under the hood explanation or please point me to resources to understand this in depth?</li>
</ol>
</blockquote>
</aside>
<p>In short, when you use  floating-point arithmetic to perform operations<br>
in different orders that should be <em>mathematically</em> equivalent, you, in<br>
general, expect to get (slightly) different results.</p>
<p>In particular, the <em>associative law</em> no longer holds, in that with<br>
floating-point arithmetic, <code>(a + b) + c  !=  a + (b + c)</code>.  (The<br>
two expressions can be equal for some values of <code>a</code>, <code>b</code>, and, <code>c</code>,<br>
but, in general, will differ.)</p>
<p>For a start, you can look at the WIkipedia article for <a href="https://en.wikipedia.org/wiki/Round-off_error" rel="noopener nofollow ugc">round-off error</a>,<br>
and dig into it more deeply, if you like, with Goldberg’s classic paper,<br>
<a href="http://perso.ens-lyon.fr/jean-michel.muller/goldberg.pdf" rel="noopener nofollow ugc">“What Every Computer Scientist Should Know About Floating-Point Arithmetic.”</a></p>
<p>Best.</p>
<p>K. Frank</p>

Vignesh_Baskaran: <p>Thank you very much for your clear explanation Frank!</p>