Manual Convolution Does not match results of pytorch Conv2d (floating point)

jianming_TONG: <p>Hi my manual implementation of the pytorch convolution2d will always have some precision difference compared with official pytorch implementation of conv2d.</p>
<p>Codes are attached for reproduce</p>
<pre><code class="lang-auto">def conv2d_matmul_fp(sample_input, weight, padding, stride, dilation):
    N,C,X,Y = sample_input.size()
    K,_,R,S = weight.size()

    out_size = (math.floor((X+padding[0]*2-dilation[0]*(R-1)-1)/stride[0]) + 1, math.floor((Y+padding[1]*2-dilation[1]*(S-1)-1)/stride[1]) + 1)

    simple_in_unfold = torch.nn.functional.unfold(sample_input, kernel_size=(R,S), dilation=dilation, padding=padding, stride=stride)
    res = torch.matmul(weight.view(weight.size()[0], -1), simple_in_unfold[0])
    return res.reshape(N, K, out_size[0], out_size[1])

def Conv2d_layer_matmul(sample_input, conv_layer):
    weight = conv_layer.state_dict()["weight"]
    padding = conv_layer.padding
    stride = conv_layer.stride
    dilation = conv_layer.dilation
    return conv2d_matmul_fp(sample_input, weight, padding, stride, dilation)

# Define sample model
sample_fp_conv2d = nn.Conv2d(3, 64, kernel_size=7, stride=(2,2), padding=(3,3), bias=False)
sample_fp_conv2d.eval()
# Extract the weights
weight = sample_fp_conv2d.state_dict()['weight']

# Define sample input data
# sample_input = val_data[0]
sample_input = torch.randn(1,3,224,224)

# Define sample result
sample_res = sample_fp_conv2d(sample_input)
print(sample_res.size())

res = Conv2d_layer_matmul(sample_input, sample_fp_conv2d)

# Comare both results
torch.where(torch.isclose(res, sample_res, rtol=1e-3)==False)
</code></pre>
<p>Really appreciate it if anyone could help</p>

KFrank: <p>Hi Jianming!</p>
<aside class="quote no-group" data-username="jianming_TONG" data-post="1" data-topic="141192" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jianming_tong/48/45606_2.png" class="avatar"> jianming_TONG:</div>
<blockquote>
<p>Hi my manual implementation of the pytorch convolution2d will always have some precision difference compared with official pytorch implementation of conv2d.</p>
</blockquote>
</aside>
<p>I have not looked at your code.</p>
<p>However, this kind of discrepancy is common and is usually caused by<br>
floating-point round-off error due to the fact that the order of operations<br>
in your implementation will almost certainly differ from those in <code>Conv2d</code>.<br>
Even though the two sets of operations should be <em>mathematically</em><br>
equivalent, the results can differ because of round-off error.</p>
<p>An easy way to test for this is to redo the calculation in double precision<br>
(preferably using the same <code>Conv2d</code> <code>weight</code> matrix and random<br>
<code>sample_input</code> tensor from your single-precision calculation, converted<br>
to double precision).  If performing the calculation in double precision<br>
reduces the discrepancy by several orders of magnitude, you would<br>
have strong evidence that your issue is due to round-off error.</p>
<p>Best.</p>
<p>K. Frank</p>

jianming_TONG: <p>Thanks for your kind reply.</p>
<p>I have tried both torch.float32 and torch.float64 for weights and input in my own implementation. And both of them demonstrate some differences compared with the official conv2d API. As for the results, double precision will even introduce more difference compared with the official implementation of Conv2d in Pytorch.</p>
<p>Might I ask whether there are other reasons except the precision issue to cause such a difference? And whether there is a custom implementation to match the custom implementation to the official Conv2d?</p>
<p>Thanks again for your help!</p>
<p>Best<br>
Jianming</p>

KFrank: <p>Hi Jianming!</p>
<aside class="quote no-group" data-username="jianming_TONG" data-post="3" data-topic="141192" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jianming_tong/48/45606_2.png" class="avatar"> jianming_TONG:</div>
<blockquote>
<p>I have tried both torch.float32 and torch.float64 for weights and input in my own implementation. And both of them demonstrate some differences compared with the official conv2d API.</p>
</blockquote>
</aside>
<p>If I understand correctly, you have used <code>torch.float32</code> and <code>torch.float64</code><br>
for your implementation, but only used <code>torch.float32</code> for the “official<br>
conv2d API.”</p>
<p>My suggestion, in more detail, is:</p>
<p>Calculation 1:</p>
<p>Use <code>torch.float32</code> for both your implementation and “the official conv2d.”<br>
Measure the discrepancy.  For example, calculate the mean and the<br>
maximum of the absolute values of the differences between the two<br>
results.</p>
<p>Calculation 2:</p>
<p>Use <code>torch.float64</code> for both your implementation and “the official<br>
conv2d.”  (Ideally perform this computation using the same <code>weight</code><br>
and <code>sample_input</code> that you used in the <code>torch.float32</code> calculation,<br>
except converted to <code>torch.float64</code>.)  The point is that you want to<br>
perform <em>both</em> convolution implementations in double precision – that<br>
is, <code>torch.float64</code>.  Measure the discrepancy the same way as you<br>
did for the <code>torch.float32</code> calculation.  Does using <code>torch.float64</code><br>
reduce the discrepancy?</p>
<p>Just to be clear, we do not expect the comparison between using<br>
<code>torch.float32</code> for one implementation and <code>torch.float64</code> for<br>
the other to reduce the discrepancy.</p>
<p>Best.</p>
<p>K. Frank</p>