Big error between tensorflow code and reproduce in pytorch

John1231983: <p>I have a tensorflow function. I want to convert it to pytorch. However, I got a big error between tensorflow and pytorch result. Could you please check my code and let me know what is the problem?</p>
<pre><code class="lang-auto">tensorflow result  0.23463342
Pytorch result 0.035501156
</code></pre>
<p>The colab is at <a href="https://colab.research.google.com/drive/1ePIEXzGWL0sp2xsW3JhVYuL2lXFEODQ7" rel="nofollow noopener">Here</a></p>
<p>This is my target tensorflow code</p>
<pre><code class="lang-auto">eps = 1e-5
ndims = 3
win = [9] * ndims
####################################
#   TENSORFLOW
####################################
def ncc_tensorflow(I, J):
  # get convolution function
  conv_fn = getattr(tf.nn, 'conv%dd' % ndims)
  # compute CC squares
  I2 = I*I
  J2 = J*J
  IJ = I*J
  # compute filters
  sum_filt = tf.ones([*win, 1, 1])
  padding = 'SAME'
  strides = [1] * (ndims + 2)
  # compute local sums via convolution
  I_sum = conv_fn(I, sum_filt, strides, padding)
  J_sum = conv_fn(J, sum_filt, strides, padding)
  I2_sum = conv_fn(I2, sum_filt, strides, padding)
  J2_sum = conv_fn(J2, sum_filt, strides, padding)
  IJ_sum = conv_fn(IJ, sum_filt, strides, padding)

  # compute cross correlation
  win_size = np.prod(win)
  u_I = I_sum/win_size
  u_J = J_sum/win_size

  cross = IJ_sum - u_J*I_sum - u_I*J_sum + u_I*u_J*win_size
  I_var = I2_sum - 2 * u_I * I_sum + u_I*u_I*win_size
  J_var = J2_sum - 2 * u_J * J_sum + u_J*u_J*win_size

  cc = cross*cross / (I_var*J_var + eps)

  # return negative cc.
  return tf.reduce_mean(cc)
</code></pre>
<p>And I reproduce it in pytorch by</p>
<pre><code class="lang-auto">####################################
#   PYTORCH
#################################### 
def ncc_torch( I, J):        
  # compute CC squares
  I2 = I*I
  J2 = J*J
  IJ = I*J
  # compute filters
  batch_size, channels, _, _, _ = I.shape
  sum_filt = torch.ones((batch_size, channels, *win)).float()
  strides = [1] * (ndims)
  # compute local sums via convolution
  I_sum = F.conv3d(I, sum_filt, stride= strides, padding=1)
  J_sum = F.conv3d(J, sum_filt, stride= strides, padding=1)
  I2_sum = F.conv3d(I2, sum_filt, stride=strides, padding=1)
  J2_sum = F.conv3d(J2, sum_filt, stride=strides, padding=1)
  IJ_sum = F.conv3d(IJ, sum_filt, stride=strides, padding=1)

  # compute cross correlation
  win_size = np.prod(win)
  u_I = I_sum / win_size
  u_J = J_sum / win_size

  cross = IJ_sum - u_J*I_sum - u_I*J_sum + u_I*u_J*win_size
  I_var = I2_sum - 2 * u_I * I_sum + u_I*u_I*win_size
  J_var = J2_sum - 2 * u_J * J_sum + u_J*u_J*win_size

  cc = cross*cross / (I_var*J_var + eps)
  
  return torch.mean(cc)
</code></pre>
<p>The unit test is</p>
<pre><code class="lang-auto"># Unit test
I = torch.rand(1,18,18,18,1) #BDHWC
J = torch.rand(1,18,18,18,1)
with tf.Session() as sess:
  tf_ncc = ncc_tensorflow(I, J)
  tf_result = sess.run(tf_ncc)
  print ('tensorflow result ', tf_result)

I = I.permute(0,4,1,2,3) #BCDHW
J = J.permute(0,4,1,2,3) #BCDHW
print('Pytorch result' , ncc_torch(I,J).numpy())
  

``</code></pre>

ptrblck: <p>Skimming through your code it looks like the padding in your PyTorch approach might be wrong.<br>
If I’m not mistaken you are using a 3D (<code>[9x9x9]</code>) conv kernel. To get the same volumetric output as your input, you should use <code>padding=4</code> for <code>stride=1</code>.</p>

John1231983: <p>Geat. It looks closer than previous using input size 2x81x81x81x1 and your kenel size suggestion</p>
<pre><code class="lang-auto">tensorflow result 0.05861372 
Pytorch result 0.029243657
</code></pre>
<p>However, the code also uses</p>
<pre><code class="lang-auto">win=[kernel size] * 3
win_size=np.prod(win)
</code></pre>
<p>So if I use kenelsize is 4, then it will be change the result on np.prod(win). Do you think we still need to keep same value win_size for both tf and pytorch, ie 9<em>9</em>9 and just modify size in the conv3d?</p>

ptrblck: <p>I’m not sure I understand the last question properly, but I would suggest to stick as close as possible to the reference implementation in order to get the same results.</p>

SimonW: <p>filter shape is wrong, should have 1st dimension of size 1</p>

John1231983: <p><a class="mention" href="/u/simonw">@SimonW</a>: Thanks. Could you show me the place that wrong? I cannot catch it</p>

Hsankesara: <aside class="quote no-group" data-username="John1231983" data-post="1" data-topic="54635">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/48/4040_2.png" class="avatar"> John1231983:</div>
<blockquote>
<p>sum_filt = torch.ones((batch_size, channels, *win)).float()</p>
</blockquote>
</aside>
<p>I think <a class="mention" href="/u/simonw">@SimonW</a> is mentioning that the above snippet would be</p>
<pre><code class="lang-auto">sum_filt = torch.ones((1, channels, *win)).float()
</code></pre>

John1231983: <p>Thanks . That is my typo. I fixed it but the result is same because my example bstch size is 1. I guess main problem is padding type</p>

John1231983: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it worked. Sorry. I mistake your comment. The padding shoud be 4 instead of kernel 4</p>