[float32 precision] Are torch.einsum and F.conv2d equivalent? Different results with torch.einsum and F.conv2d

dluca12: <p>I need to use <code>torch.einsum</code> to do the convolution instead of using <code>torch.nn.functional.conv2d</code>.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/books.png?v=12" title=":books:" class="emoji" alt=":books:" loading="lazy" width="20" height="20"> In short, I got 16 images with the size of (3, 4, 4) and 16 kernels with the size of (3, 4, 4). I need to apply one kernel to one corresponding image and get one output (one data point).</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

torch.manual_seed(1)
weight = torch.normal(10**-5, 10**-6, (16, 3, 4, 4))
bias = torch.normal(10**-5, 10**-6, (16,1))

img = torch.randn(16, 3, 4, 4)
</code></pre>
<p>I compared the convolutional operation with <code>einsum</code> and <code>conv2d</code>:</p>
<p>With</p>
<pre><code class="lang-auto">F.conv2d(img[0], weight[0].unsqueeze(0), bias[0]).item()
</code></pre>
<p>I got <code>4.617268132278696e-05</code></p>
<p>But with</p>
<pre><code class="lang-auto">(torch.einsum('chw,chw-&gt;', [img[0], weight[0]]) + bias[0]).item()
</code></pre>
<p>I got <code>4.617268859874457e-05</code>.</p>
<p>All data <code>dtype = torch.float32</code>.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=12" title=":question:" class="emoji" alt=":question:" loading="lazy" width="20" height="20"> <strong>I would like to know whether <code>einsum</code> and <code>conv2d</code> are equivalent in my scenario.</strong></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/bulb.png?v=12" title=":bulb:" class="emoji" alt=":bulb:" loading="lazy" width="20" height="20"> The reason of implementing with <code>torch.einsum</code>:</p>
<p>I have 16 images and 16 kernels and need to applying one kernel to one image to get one output. It is easy to directly get all outputs (for 16 images) with</p>
<pre><code class="lang-auto">torch.einsum('bchw,bchw-&gt;b', [img, weight])+bias.squeeze(-1)
</code></pre>
<p>The output:</p>
<pre><code class="lang-auto">tensor([ 4.6173e-05, -9.3411e-06, -8.0316e-05, -6.5993e-05,  1.3381e-04,
        -2.3025e-05, -1.3640e-06,  9.6504e-05,  2.1309e-06, -4.2717e-05,
         3.5023e-06,  3.2773e-05,  2.0304e-04, -2.4030e-05,  1.0894e-04,
         6.0090e-05])
</code></pre>

KFrank: <p>Hi Luca!</p>
<aside class="quote no-group" data-username="dluca12" data-post="1" data-topic="166829" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dluca12/48/55146_2.png" class="avatar"> dluca12:</div>
<blockquote>
<p>With</p>
<pre><code class="lang-auto">F.conv2d(img[0], weight[0].unsqueeze(0), bias[0]).item()
</code></pre>
<p>I got <code>4.617268132278696e-05</code></p>
<p>But with</p>
<pre><code class="lang-auto">(torch.einsum('chw,chw-&gt;', [img[0], weight[0]]) + bias[0]).item()
</code></pre>
<p>I got <code>4.617268859874457e-05</code>.</p>
</blockquote>
</aside>
<p>For this computation, <code>conv2d()</code> and <code>einsum()</code> are equivalent.  The<br>
difference* you see is due to numerical round-off error (because the two<br>
versions are performing the computations in different, but mathematically<br>
equivalent orders).</p>
<p>Your two results agree to seven decimal digits, which is what you would<br>
expect with single-precision (<code>float32</code>) accuracy.</p>
<aside class="quote no-group">
<blockquote>
<p>I have 16 images and 16 kernels and need to applying one kernel to one image to get one output. It is easy to directly get all outputs (for 16 images) with</p>
<pre><code class="lang-auto">torch.einsum('bchw,bchw-&gt;b', [img, weight])+bias.squeeze(-1)
</code></pre>
<p>The output:</p>
<pre><code class="lang-auto">tensor([ 4.6173e-05, -9.3411e-06, -8.0316e-05, -6.5993e-05,  1.3381e-04,
        -2.3025e-05, -1.3640e-06,  9.6504e-05,  2.1309e-06, -4.2717e-05,
         3.5023e-06,  3.2773e-05,  2.0304e-04, -2.4030e-05,  1.0894e-04,
         6.0090e-05])
</code></pre>
</blockquote>
</aside>
<p><code>einsum()</code> is a perfectly fine way to perform your computation.  Note,<br>
however, that you can use <code>conv2d()</code> for this if you merge your “batch”<br>
and channels dimensions together and use <code>conv2d()</code>'s <code>groups</code> feature:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed(1)
&gt;&gt;&gt; weight = torch.normal(10**-5, 10**-6, (16, 3, 4, 4))
&gt;&gt;&gt; bias = torch.normal(10**-5, 10**-6, (16,1))
&gt;&gt;&gt;
&gt;&gt;&gt; img = torch.randn(16, 3, 4, 4)
&gt;&gt;&gt;
&gt;&gt;&gt; result = torch.einsum('bchw,bchw-&gt;b', [img, weight])+bias.squeeze(-1)
&gt;&gt;&gt; resultB = torch.nn.functional.conv2d (img.reshape (48, 4, 4), weight, bias.squeeze(), groups = 16).squeeze()
&gt;&gt;&gt;
&gt;&gt;&gt; torch.allclose (result, resultB)      # equal up to round-off error
True
&gt;&gt;&gt; torch.equal (result[0], resultB[0])   # but first elements happen to be equal
True
&gt;&gt;&gt;
&gt;&gt;&gt; resultB
tensor([ 4.6173e-05, -9.3411e-06, -8.0316e-05, -6.5993e-05,  1.3381e-04,
        -2.3025e-05, -1.3640e-06,  9.6504e-05,  2.1309e-06, -4.2717e-05,
         3.5023e-06,  3.2773e-05,  2.0304e-04, -2.4030e-05,  1.0894e-04,
         6.0090e-05])
</code></pre>
<p>*)  Oddly enough, I get the <em>same</em> results for <code>conv2d()</code> and <code>einsum()</code> when<br>
I run your code.  This happens on a couple of different versions of pytorch<br>
(including 1.13.0) on both windows and linux.</p>
<p>Best.</p>
<p>K. Frank</p>

dluca12: <p>Hi Frank,</p>
<p>Thank you so much for your vivid and clear explanation!</p>
<p>FYI, I am using macOS with an M1 chip with torch = 1.12.1.</p>
<p>Best wishes,<br>
Luca</p>